{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 第4章 前馈神经网络\n",
    "\n",
    "神经网络是由神经元按照一定的连接结构组合而成的网络。神经网络可以看作一个函数，通过简单非线性函数的多次复合，实现输入空间到输出空间的复杂映射 。\n",
    "前馈神经网络是最早发明的简单人工神经网络。整个网络中的信息单向传播，可以用一个有向无环路图表示，这种网络结构简单，易于实现。\n",
    "\n",
    "在学习本章内容前，建议先阅读《神经网络与深度学习》第2章：机器学习概述的相关内容，关键知识点如 **图4.1** 所示，以便更好的理解和掌握相应的理论知识，及其在实践中的应用方法。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/0ae8775e92a04173928b4e3fed98b5934f0aad6582dc4143b2205669bfae6b55\" width=500></center>\n",
    "<br><center>图4.1 《神经网络与深度学习》关键知识点回顾</center></br>\n",
    "\n",
    "本实践基于 **《神经网络与深度学习》第4章：前馈神经网络** 相关内容进行设计，主要包含两部分：\n",
    "\n",
    "* **模型解读**：介绍前馈神经网络的基本概念、网络结构及代码实现，利用前馈神经网络完成一个分类任务，并通过两个简单的实验，观察前馈神经网络的梯度消失问题和死亡ReLU问题，以及对应的优化策略；\n",
    "* **案例与实践**：基于前馈神经网络完成鸢尾花分类任务。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.1 神经元\n",
    "\n",
    "神经网络的基本组成单元为带有非线性激活函数的神经元，其结构如如**图4.2**所示。神经元是对生物神经元的结构和特性的一种简化建模，接收一组输入信号并产生输出。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/d084b5e9e8654367ab4af046184f2f294f85a48417e64e36a79df942d5d6ff68\" width=\"400\" hegiht=\"\" ></center>\n",
    "<center><br>图4.2: 典型的神经元结构</br></center>\n",
    "\n",
    "### 4.1.1 净活性值\n",
    "\n",
    "假设一个神经元接收的输入为$\\mathbf{x}\\in \\mathbb{R}^D$，其权重向量为$\\mathbf{w}\\in \\mathbb{R}^D$，神经元所获得的输入信号，即净活性值$z$的计算方法为\n",
    "\n",
    "$$\n",
    "z =\\mathbf{w}^T\\mathbf{x}+b，（4.1）\n",
    "$$\n",
    "\n",
    "其中$b$为偏置。\n",
    "\n",
    "为了提高预测样本的效率，我们通常会将$N$个样本归为一组进行成批地预测。\n",
    "\n",
    "$$\n",
    "\\boldsymbol{z} =\\boldsymbol{X} \\boldsymbol{w} + b, (4.2)\n",
    "$$\n",
    "\n",
    "其中$\\boldsymbol{X}\\in \\mathbb{R}^{N\\times D}$为$N$个样本的特征矩阵，$\\boldsymbol{z}\\in \\mathbb{R}^N$为$N$个预测值组成的列向量。\n",
    "\n",
    "使用Paddle计算一组输入的净活性值。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input X: Tensor(shape=[2, 5], dtype=float32, place=CPUPlace, stop_gradient=True,\n",
      "       [[0.70939004, 0.77428746, 0.13596979, 0.59024960, 0.73392862],\n",
      "        [0.73414218, 0.48377231, 0.14179315, 0.41931525, 0.70087898]])\n",
      "weight w: Tensor(shape=[5, 1], dtype=float32, place=CPUPlace, stop_gradient=True,\n",
      "       [[0.46766225],\n",
      "        [0.77609837],\n",
      "        [0.25065762],\n",
      "        [0.72651631],\n",
      "        [0.87527806]]) \n",
      "bias b: Tensor(shape=[1, 1], dtype=float32, place=CPUPlace, stop_gradient=True,\n",
      "       [[0.01581415]])\n",
      "output z: Tensor(shape=[2, 1], dtype=float32, place=CPUPlace, stop_gradient=True,\n",
      "       [[2.05379176],\n",
      "        [1.68824458]])\n"
     ]
    }
   ],
   "source": [
    "import paddle\r\n",
    "\r\n",
    "# 2个特征数为5的样本\r\n",
    "X = paddle.rand(shape=[2, 5])\r\n",
    "\r\n",
    "# 含有5个参数的权重向量\r\n",
    "w = paddle.rand(shape=[5, 1])\r\n",
    "# 偏置项\r\n",
    "b = paddle.rand(shape=[1, 1])\r\n",
    "\r\n",
    "# 使用'paddle.matmul'实现矩阵相乘\r\n",
    "z = paddle.matmul(X, w) + b\r\n",
    "print(\"input X:\", X)\r\n",
    "print(\"weight w:\", w, \"\\nbias b:\", b)\r\n",
    "print(\"output z:\", z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**********\n",
    "\n",
    "**说明**\n",
    "\n",
    "在飞桨中，可以使用**nn.Linear**完成输入张量的上述变换。\n",
    "\n",
    "**********"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.1.2 激活函数\n",
    "\n",
    "净活性值$z$再经过一个非线性函数$f(·)$后，得到神经元的活性值$a$。\n",
    "\n",
    "$$\n",
    "a = f(z)，（4.3）\n",
    "$$\n",
    "\n",
    "激活函数通常为非线性函数，可以增强神经网络的表示能力和学习能力。常用的激活函数有S型函数和ReLU函数。\n",
    "\n",
    "#### 4.1.2.1 Sigmoid 型函数\n",
    "\n",
    "Sigmoid 型函数是指一类S型曲线函数，为两端饱和函数。常用的 Sigmoid 型函数有 Logistic 函数和 Tanh 函数，其数学表达式为\n",
    "\n",
    "\n",
    "Logistic 函数：\n",
    "\n",
    "$$\n",
    "\\sigma(z) = \\frac{1}{1+\\exp(-z)}。（4.4）\n",
    "$$\n",
    "\n",
    "Tanh 函数：\n",
    "\n",
    "$$\n",
    "\\mathrm{tanh}(z) = \\frac{\\exp(z)-\\exp(-z)}{\\exp(z)+\\exp(-z)}。（4.5）\n",
    "$$\n",
    "\n",
    "Logistic函数和Tanh函数的代码实现和可视化如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADuCAYAAAAOR30qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VNX5+PHPnSX7MlkICRO2LEQIhAgTQWstW0BcoiJV+FHFIo21Wiv61dJWUOsCVWtti/X7jRvUItFaWWoxLCquCLJpZdGwBLMRs5J19vP7IzIQkkACmWQSn/frFcjcc+69z50MDyfnnnuOppRCCCFEz9P1dABCCCGaSUIWQggfIQlZCCF8hCRkIYTwEZKQhRDCR0hCFkIIHyEJWQghfIQkZCGE8BGSkIUQwkcYOllfHusTPunyyy8nLy+vp8MQoj1aRypJC1n0CRUVFT0dghDnTRKyEEL4CEnIQgjhIyQhCyGEj5CELIQQPkISshBC+AhJyMIr5s2bR0xMDCNHjmyzXCnFXXfdRVJSEmlpaezatctTtmLFCpKTk0lOTmbFihXdFbIQPU4SsvCKW2655Yzjgt9++23y8/PJz88nJyeH22+/HYCqqioefvhhtm3bxvbt23n44Yeprq7urrC/t5RSnLp60InXp391RXl7Zb5U7na7W311x+pKnX0wRIgOueyyyygoKGi3fO3atdx8881omsb48eOpqamhtLSULVu2kJmZSWRkJACZmZnk5eUxe/bsborcO5TLjbvehj48EABncTWub2tRdifK6UY5XWiaRuBlKQBYPzuCs7ga3G5QoNwKXaCR4KvSAWh8bz/OompOfVZLFxZIyDVjAGjI+wJXWe13J2/+Qwv0x69fNO5aG46aquZjn/K4gmp0w2ErqsmBNjIELVDf4hrcxY043y9Dudz4zRyMFtgyfTj3VWNfWwBON4ELL0QztmzvObaVNZejCH58fKv3yP5uEfZ1BeCvI+QPl7Qqt607gv2tAjSTPyFPtC63vpaPY3Mhutgggh9pfXzr8v04PixBNySM4EUZrcqbnvsvzs++RX9BBEH3j2lR1vj0bvp/djsEGFvt15UkIYseUVxczMCBAz2v4+PjKS4ubnd7W3JycsjJyQGgvLzcuwF3kvPYcZxHK3BVNeCus6IabACEzr0UTafhOFqB46tjYNSjGfRg0KH5n/zHrgUY0YUGgE4DTUPTaWj+zf9c3Y12NE2Pzs8fVWdHNTpQjQ4ce49T/sxy3GX16BJDIcgALnfzARWoaiu17zW/l8ZJ8Wjhfs3J+ruWn7vSimt3BVqgEeO3AyDUD02vNceg01C1DlSDHQw6XPtqwE8PuubY0GloVvCzDAC9DndhE5oOQPMkfV1YEAFXDwNNw13UBNp3BVrzH4ZBURh+amo+V5ndsx2t+S+/C834DR/QXF7pPPlefXeYgB8MIcAyuPn3/ipXyx+IBgGZSQT8KAn0QPVp5UDgNcPhiuHNWbHGxan/WwXNSgOD9zsUJCGLXis7O5vs7GwALBZLj8Wh3ApXSTWOo5X4jx2CLsCIq7wO+9fH0EcEY4gNRxcagBYaQHMG1AjISCBgfCKaru1/5H6pA3CFBuP48lucX37b/PfBKqoL3sRd1tCqvhbuj75/CLqYYAwj+qGLDkJnCkAX5o8WfuJvf3SL/dHC/NFC/NACDGiBxu/+NjT/xyB6lCRk0SPMZjOFhYWe10VFRZjNZsxmM1u2bGmxfcKECd0fYAcomwP7vhLsXx9DNdrBqMeYGIMuNhy/4XH4pZqbW49t0Iwtk5+rvAH7p0XYtxZi31qEY0cJqt7uKdcPDMMwLArj1SkYhpjQDzWhH2JCPzAcfUywp/Usejf5KYoekZWVxbJly5g1axbbtm0jPDycuLg4pk2bxm9/+1vPjbyNGzeyZMmSHo62NXejnfo3d4DDhd4cgd+4RAwDI9H0zS3es7U2ld2J/eNCrHkHseYdxPlFWXOBQYfxwliCbknHmNYfw8gYjCP6oQsP8PYlCR8gCVl4xezZs9myZQsVFRXEx8fz8MMP43A4APj5z3/OFVdcwfr160lKSiIoKIiXX34ZgMjISBYtWkRGRvNNl8WLF3tu8PkCd6MdXZAfuiA//NMGYoiPRB8Z3KF9lcOFbfNhGnO/xLp6P6rODkYdfpcOImzJZPwuHYTf2AFogd69cSR8l9bJoRwy/abwSRaLhR07dnjt+Eop7PtLsO0oIPjqdPQRHUvCAM78Sur/9hlNr3yOu7IJzRRA4IzhBFx7Af4Th6AL8fda3MJndGj6TWkhC3EWSils2w5j31+CYWAkuuCzJ1ClFLZNh6j/41ZsGw+BQUfgjOEE/iSNgKmJ0ucr2iSfCiHOQCmFdetBHF8dwy/VjH/GUDSt/caOUgrb2/nU/v59HNuK0ZlDCf39RIJ/NgZ9bGg3Ri56I0nIQpyB80h5czJOG4j/mMFnTMb2nSUc/9Xb2D8uRD/EhCnnaoLmjkbzk39momPkkyLEGRiG9iNQ0zAMiW43GbvKG6j9zWYaX9qNrl8wpv+7iqCfXthqaJsQZyMJWYg2uOutoNPQBfljHNqv3XpN/9pHze1v4a6xEnLvJYQ+cJkMURPnTBKyEKdRbkXT+1/hbrITMsPS5sMd7uomam5/i6bX9mIcG0f0e7dgTI3pgWhFXyIJWYjTOA6U4vq2loDLUtpMxvadJVTNfB1XUS2hj0wk9NeXSveE6BKSkIU4hdtqx7r7KPo4E8aE1l0VDS/spObO9ehjgun34U/xGz+wjaMIcW4kIQtxCtuuo+BwETA+ocVNPOV2U3vfJuqf3or/1EQiVs5AH93xh0OE6AhJyEJ8RykFTjd+F8ShN51MtsrqoOqm1Vjf2EfwLy8i/E+Xe+asEKIrSUIW4jsnJog/dToBd52NyitXYv/oG8Kfnkbw3ePPOBZZiPMhCVkIwN1gQ9md6COCPQnXfdxK5fR/YP+shIhVMwm6se31AYXoKvJ7lxCAfV8xDet2o2zNM9K5a5qomPoK9s9KiHz9x5KMRbeQFrL43lNOF/avyzAMikLzN+JutFN5xUocu0uJ/NcNBGZd0NMhiu8JScjie89xuBzsTvyGx6EcLqpu+Cf2bcVE/vPHkoxFt5IuC+EVeXl5pKSkkJSUxNKlS1uVL1iwgPT0dNLT0xk2bBgmk8lTptfrPWVZWVlej9V+oBSdKQhdTBg1P1uH7T/5mP52JYEzRnj93EKcSlrIosu5XC7uuOMONm3aRHx8PBkZGWRlZTFixMkE96c//cnz/V//+ld2797teR0YGMiePXu6JVZ3ow13TSP+Y4dQ/+AWGld8TujDEwi+recWTRXfX9JCFl1u+/btJCUlkZCQgJ+fH7NmzWLt2rXt1l+1ahWzZ8/uxghP0gX5EzprHM7/VlH36AcE3XohoYt+1COxCCEJWXS54uJiBg48+UhxfHw8xcXFbdY9evQoR44cYdKkSZ5tVqsVi8XC+PHjWbNmTbvnycnJwWKxYLFYKC8vP+d4HV9+y/F5a/G7dBCmv10p44xFj5EuC9GjcnNzmTlzJnr9ycl5jh49itls5vDhw0yaNIlRo0aRmJjYat/s7Gyys7OB5jX1OstZdhzrxwdp+P02dNFBRP7rBplMXvQoaSGLLmc2myksLPS8Lioqwmw2t1k3Nze3VXfFiboJCQlMmDChRf9yV3IcLsdVWY/raA2Ra2ejjwnxynmE6ChJyKLLZWRkkJ+fz5EjR7Db7eTm5rY5WuLAgQNUV1dz8cUXe7ZVV1djs9kAqKio4OOPP25xM7CrKKVwfFmC87+VhD9zBX4XxnX5OYToLPn9THQ5g8HAsmXLmDZtGi6Xi3nz5pGamsrixYuxWCye5Jybm8usWbNa9Nnu37+f2267DZ1Oh9vtZuHChV5JyNYNX4FBQx8UQPDc9C4/vhDnQjt1IpUO6FRlIbqLxWJhx44dHarrKqun+p5/Y5xsJjhrDIZo6aoQXtehO8XSZSG+V5RSVM9djetQDcbYSEnGwqdIQhbfKw3PfYZtwyFCbrEQdIVMGCR8iyRk8b3hzK+k9r5N+F9/AQE3pfV0OEK0IglZfC8ol5vquavBT0/Q/JE0rt1FJ++fCOF1MspCfC/UP/kx9q1FmFbOwHW8AX2cSZ7IEz5HWsiiz3N8cYzaxe8R+OMR+E9PRDXaMQwwnX1HIbqZJGTRpymXm+p5a9FFBhL+tytxlx4HkIQsfJIkZNGnNSzbjmNnKeF/mY4+OhhnaQ1aSAC60MCeDk2IVqQPWfRZzsLj1D7wLv7Tkwj8cSoAARcl4G6093BkQrRNErLos47/cj243C2m1NSFBqALDejhyIRom3RZiD6pafV+rGu/IvThiRiGRADgPFaD/UApyuXu4eiEaJskZNHnuGut1PxyPcbR/Qm5e7xnu+Pgt9h2FYBOhrsJ3yRdFqLPqX3gXdwldUS9eSOa8eTE966yWvQxYTL+WPgsaSGLPsW+vYiGZdsJvuMi/C6K92x3W+24a5vQx4T1YHRCnJkkZNFnKKeLmux/o4sLJeyxSS3KXGV1AOj7S0IWvku6LESfUf/Mpzg+LyPyXzegC2s5ksJ9vBH0Gvqo0B6KToizkxay8Jq8vDxSUlJISkpi6dKlrcqXL19Ov379SE9PJz09nRdeeMFTtmLFCpKTk0lOTmbFihVnPZeyu6h7cAsBWSkEXDe8Vbl/2kBCZ1+MZpCPvPBdsmKI8AqXy8WwYcPYtGkT8fHxZGRksGrVqhbLMS1fvpwdO3awbNmyFvtWVVV5VgDRNI2xY8eyc+dOIiIi2jyXUooLTYmsd91KzL47MAySx6KFz5EVQ0TP2b59O0lJSSQkJODn58esWbNYu3Zth/bdsGEDmZmZREZGEhERQWZmJnl5ee3Wb/rnXlStlbBHJ7WZjN11TTRu+hJXRf05X48Q3UESsvCK4uJiBg4c6HkdHx9PcXFxq3r/+te/SEtLY+bMmRQWFnZq35ycHH6UPp5Dc/6By08j+M6L2ozFVV6Hs6haPu3C58lHVPSYq6++moKCAr744gsyMzOZO3dup/bPzs5m7cUPEOkOxD8xGs2gb7Oeq6Ie9Dp0pqCuCFsIr5GELLzCbDZ7WrwARUVFmM3mFnWioqLw9/cHYP78+ezcubPD+wLYPvmGhv/dQfBd49CCjO3G4qqoQx8ZjKaTj7vwbfIJFV6RkZFBfn4+R44cwW63k5ubS1ZWVos6paWlnu/XrVvH8OHNoyOmTZvGxo0bqa6uprq6mo0bNzJt2rQW+yq7k5rb3kI/MIywRya2G4dSCldlA7ooWV1a+D4Zhyy8wmAwsGzZMqZNm4bL5WLevHmkpqayePFiLBYLWVlZ/OUvf2HdunUYDAYiIyNZvnw5AJGRkSxatIiMjAwAFi9eTGRkZIvj1z/1Cc4vvyVy3Wx0If7txqFsTvRRwRjkgRDRC8iwN9HrOPMrKRv1NwKuTiHqnzcAeIbJCeGjZNib6HuUUtT8/C00fwOmP1/eofpC9BaSkEWv0vj3z7G9e4TwP0xBP+Ds3RBNm/fR9MFX3RCZEOdPErLoNVzlDRy/ZwN+lwwkKHvsWesrpXCV18r8x6LXkIQseo3j925A1dkw5VzdoSFsqsmBsjnRRQZ3Q3RCnD9JyKJXsG46RNMrXxD660sxpsZ0aB93dQMA+ghJyKJ3kIQsfJ673kZN9r/RJ0cS+rsfdng/V1VzQtZJQha9hIxDFj6v9v5NuI7WEP3hPLSA9p/IO50uPBDjsFh0ndhHiJ4kCVn4NOvmQzQ8t4OQey7G/weDOrWvcVAUxkFRXopMiK4nXRbCZ7lrrdTcug7DsCjCHp109h1OodwKd5PdS5EJ4R2SkIXPOn7fJlxFtUSsuBYtsHPdDu7jjdTnbsNxuNxL0QnR9SQhC59kzcunMWcnIfdejN/4gWff4TQnRljIlJuiN5GELHyOq6ye6rlrMIyMIez37c/kdsZjVDWATkMXHtjF0QnhPXJTT/gU5XZTPXc17lob0e/c3KlRFadyVzegCw9E00ubQ/Qe8mkVPqX+mU+xbThE+B+nYhzZ/5yP46pplPHHoteRFrLwGfZdJdQu3EzANSkE355xXsfyTx+MLtiviyITontIQhY+wV3VSNXM19H1C8b0Qhaadn4TAvkln3vrWoieIl0Wwivy8vJISUkhKSmJpUuXtip/+umnGTFiBGlpaUyZNJni61fiKqol8o0b8OsfRnp6Ounp6a2WfeoId70VV2U9yi1zIYveRVrIosu5XC7uuOMONm3aRHx8PBkZGWRlZTFixAhPnQsvvJAdO3YQFBTEu1c+iW59MabnrsT/4oEEBgayZ8+ecz6/Pb8M++ffEHrTD+jgQg1C+ARpIYsut337dpKSkkhISMDPz49Zs2axdu3aFnUmTpxIUFAQTesOkLK+gfdiigm6zdIl53fXNKILCZARFqLXkU+s6HLFxcUMHHjyYY74+HiKi4tb1bPvLKF69r8ojXFQ+PNBnn5jq9WKxWJh/PjxrFmzpt3z5OTkYLFYsFgslJeffCLPfbxRHggRvZJ0WYge4fymhsqrXqUxSPGA+UPW/naTp+zo0aOYzWYOHz7MpEmTGDVqFImJia2OkZ2dTXZ2NtC8yCl8N4dFbRMGc0T3XIgQXUhayKLLmc1mCgsLPa+Lioowm82e1+6aJiqvWImjrok7Q//N3/PewN/fv8X+AAkJCUyYMIHdu3d3+Nyq3gouhS5cWsii95GELLpcRkYG+fn5HDlyBLvdTm5urme0hLveRuWVr+L4qoJ7w/L464Z/EBNzcgWQ6upqbDYbABUVFXz88cctbgaejRbkR9DUkdJCFr2SdFmILmcwGFi2bBnTpk3D5XIxb948UlNTefg3i7nxrTBC9zewbMQ+Pig7yI9//GMABg0axLp169i/fz+33XYbOp0Ot9vNwoULO5eQDXpJxqLX0pTq1FhNGdgpzomyOqi87jVsGw4S8coMguakdenxLRYLO3bswFFUhaZpkpSFr+nQ+EtpIQuvc9daqbw2F/t7BZiev7rLk/Gp7J8XgoYkZNErSUIWXuUqb6By+j9wfF5GxD+6vmV8KqUUrppGjEOjvXYOIbxJErLwGsfXFVRdvQrnN8eJWjOLgCuHefV8yuoAu1NGWIheSxKy8AprXj5Vs95AM+qJ3nxzpxcoPRfu440A6OWhENFLybA30aWU203d0g+pvPJVDENM9NuR3S3JGJrHNwOySojotaSFLLqMq7iW6ptXY3v3CIE3pGJ66ZpunZPYmNwffWw4WrD/2SsL4YMkIYvzppSiKfdLjv9yParJiemFLILmXXjecxp3lqbXSXeF6NUkIYvz4jxURc0v/oNt4yGMGQOIeGUGxpSeGeVg3VWAITYcwwAZ8iZ6J0nI4py4a5qo+8PH1D/zKZpRR/hfpxN8e0bPTXmpmscgazqdJGTRa0lCFp3irrfR8NwO6pZ8iKq2Evj/RhH+RCZ6c1jPBuZyA6AzyQ090XtJQhYd4iqto/6v22h4bgeqxor/5UmELZmMX3pcT4cGgDqRkGUMsujFJCGLdimXG9vGQzQu30PTmgPgcBEwYzgh916C/8UDz36A7uRygwa6MGkhi95LErJoQTld2D8upGntAZpe24u7pA5dVCDBP7cQ8suLMCRF9XSIbVJuhS40UJZtEr2aJGSB82gN9vcLsL5zBNt/vsZd2QR+egKmJRL01+kEXDUMzc+3Pyq6EH+CrxnT02EIcV58+1+Z6HLuWiuOz8tw7C7FvqME+wdHcR09DoAWGUjAFckEXpOC/7QkdKG96wELzSCtY9G7SULug5TLjauoFufBKlwHq3AerMKZX4njy29xHar21NPFBON36SBC7rkY/x8NwTAqBk3XdUktLy+PX/3qV7hcLubPn8/ChQtblNtsNm6++WZ27txJVFQUr732GkOGDAFgyZIlvPjii+j1ev7yl78wbdq0ds/jrrfirrPiqqhDHx3aZfEL0d0kIfcCyu7EXWPFXWNFffe3u9qKqm7CVdaAq6QOd2kdrtL65u/L6sF1yloCAQYMCREY02MJ/umFGNNjMV4Yiy4u1GtP07lcLu644w42bdpEfHw8GRkZZGVltVj948UXXyQiIoKDBw+Sm5vLr3/9a1577TX27dtHbm4ue/fupaSkhClTpvD111+j1+vbPld1A8rmRLll/QTRu3VLQnZ8XQFWZ/ML1fyo7Ynvm/9WHfv+xF+n7t/e923sr0471hn3P+1Y6tRjuRU43c1DrZzNX6qdvz3fu9TJMocLrE5UkxPV6EA1OZq/P/Xvxu++r7ejGh3tv7ka6PoFo48LQRcXijGtP/q4EPRDTBiSIjEkRaIbENqlLd+O2L59O0lJSSQkJAAwa9Ys1q5d2yIhr127loceegiAmTNncuedd6KUYu3atcyaNQt/f3+GDh1KUlIS27dv5+KLL27zXCcmFdLLpEKil+uWhFx13Ws495V3x6l8m0EHBh2aQYcWYEALNKAFGdECjc3fBxrRRQSiBRrgu226EL/mbaYAdBEB6EwBzd+f+OoXjGZsu+XYk4qLixk48OTQuPj4eLZt29ZuHYPBQHh4OJWVlRQXFzN+/PgW+xYXF7d7LvfxRjSdhuZv7OKrEKJ7dWpNvcsvv1xVVFR0+iSq1sbx6uOEhZ/laS6tzW9Pvjrbb9ftlre/f01NDSaTqfP7a4Cmtfhea7FN+277Kds6oby8nH79+nV6P2/raFzV1dXU1tYyePBgACorK2loaGDQoJNTce7du5fk5GT8/JpnhPvvf//L8OHDKSkpISQkhMjISAAKCgoIDw8nIqLlI9Hl5eVUVFQwyBRDQXkJqWmjuuoyu0xv/zl2t74a186dOzcopS4/a0WlVGe+ztnYsWPPZ3evkbg6p6NxffLJJ2rq1Kme148//rh6/PHHW9SZOnWq+uSTT5RSSjkcDhUVFaXcbneruqfWa0v9hv+qQf3iOnMZ3aa3/xy7Wx+Oq0M5VsYJCa/IyMggPz+fI0eOYLfbyc3NJSsrq0WdrKwsVqxYAcAbb7zBpEmT0DSNrKwscnNzsdlsHDlyhPz8fC666KJ2zxU8dSQVDce9ej1CdAcZZSG8wmAwsGzZMqZNm4bL5WLevHmkpqayePFiLBYLWVlZ3Hrrrdx0000kJSURGRlJbm4uAKmpqdxwww2MGDECg8HAs88+2+4ICyH6lI42pdV5dln83//93/ns7jUSV+f4Wly2r4+punW71OBBg3s6lDb52vt1gsTVOV0QV4dybKdu6nFy4JcQPsG67RD2r48x6c+/YseOHT0djhDt6dCdfelDFr2aq6ZRptwUfUaXJuR//vOfpKamotPpWrVWlixZQlJSEikpKWzYsKHN/Y8cOcK4ceNISkrixhtvxG63d2V4ANx4442kp6eTnp7OkCFDSE9Pb7PekCFDGDVqFOnp6Vgsli6P43QPPfQQZrPZE9v69evbrJeXl0dKSgpJSUksXbrU63Hdd999XHDBBaSlpXHddddRU1PTZr3uer9Ov3738aYWD4TYbDZuvPFGkpKSGDduHAUFBV6L5YTCwkImTpzIiBEjSE1N5c9//nOrOlu2bCE8PNzz8/3973/v9bjg7D8XpRR33XUXSUlJpKWlsWvXLq/H9NVXX3neh/T0dMLCwnjmmWda1Omu92vevHnExMQwcuRIz7aqqioyMzNJTk4mMzOT6urqNvddsWIFycnJJCcne25On7eO9m2oDvQh79u3Tx04cED96Ec/Up999pln+969e1VaWpqyWq3q8OHDKiEhQTmdzlb7//jHP1arVq1SSil12223qb/97W/n1FnTUffcc496+OGH2ywbPHiwKi8v9+r5T/Xggw+qJ5988ox1nE6nSkhIUIcOHVI2m02lpaWpvXv3ejWuDRs2KIfDoZRS6v7771f3339/m/W64/06/fovGmNRx1/6QFn3HPUMS3r22WfVbbfdppRSatWqVeqGG27wakxKKVVSUqJ27typlFKqtrZWJScnt/q5vPfee+rKK6/0eiynO9vP5T//+Y+6/PLLldvtVlu3blUXXXRRN0bX/DPt37+/KigoaLG9u96v999/X+3cuVOlpqZ6tt13331qyZIlSimllixZ0uZnvrKyUg0dOlRVVlaqqqoqNXToUFVVVXWmU3X/sLfhw4eTkpLSant7j8Ke/h/Du+++y8yZMwGYO3cua9as6crwWp3v9ddfZ/bs2V47R1c79XFkPz8/z+PI3jR16lQMhubBOOPHj6eoqMir5zuT06///91wIwetFej7nZxQaO3atcydOxdofhz7nXfeOfmovpfExcUxZkzz1J+hoaEMHz78jE8W+pK1a9dy8803o2ka48ePp6amhtLS0m47/zvvvENiYqLnAaLudtlll3keQDrh1M9Qe3low4YNZGZmEhkZSUREBJmZmeTl5Z13PN3Sh9zWY7Snf2ArKysxmUyef/xne1z2fH344Yf079+f5OTkNss1TWPq1KmMHTuWnJwcr8VxqmXLlpGWlsa8efPa/DWpI++jN7300ktMnz69zbLueL9Ov/7IAf15ae87LRY1be9x7O5SUFDA7t27GTduXKuyrVu3Mnr0aKZPn87evXu7JZ6z/Vx6+jOVm5vbbqOoJ94vgLKyMuLimpcmi42NpaysrFUdb71vnR6HPGXKFI4dO9Zq+2OPPcY111xz3gF1hY7EuGrVqjO2jj/66CPMZjPffvstmZmZXHDBBVx22WVei+v2229n0aJFaJrGokWLuPfee3nppZfO63xdEdeJ9+uxxx7DYDAwZ86cNo/hjffrbDQfG/NTX1/P9ddfzzPPPENYWMtpAsaMGcPRo0cJCQlh/fr1XHvtteTn53s9pp74uXSU3W5n3bp1LFmypFVZT71fp9M0zWszIral0wl58+bNnT6J2WymsLDQ87qoqAiz2dyiTlRUFDU1NTidTgwGQ5t1uipGp9PJm2++yc6dO88YM0BMTAzXXXcd27dvP+8Pckffu5/97GdcddVVbcZ0tvfRG3EtX76ct956i3feeafdD6c33q+2znHq9ac2hGAZdXmbdeLj43E6nRw/fpyoKO8vO+XrlaqQAAAbfElEQVRwOLj++uuZM2cOM2bMaFV+aoK+4oor+MUvfkFFRQXR0dFejetsPxdvfaY64u2332bMmDH079+/VVlPvV8A/fv3p7S0lLi4OEpLS4mJiWlVx2w2s2XLFs/roqIiJkyYcN7n7pYui448CqtpGhMnTuSNN94Amu9geqvFvXnzZi644ALi4+PbLG9oaKCurs7z/caNG1vchfWGU/vtVq9e3eb5OvI4clfLy8vjiSeeYN26dQQFtT28rLver9OvX9/gwNS/5T/Q9h7H9ialFLfeeivDhw/nnnvuabPOsWPHPH3Z27dvx+12e/0/io78XLKysvj73/+OUopPP/2U8PBwz6/r3nam31J74v064dTPUHt5aNq0aWzcuJHq6mqqq6vZuHHjGRdR6LCO3v1THRhl8eabbyqz2az8/PxUTExMi8llHn30UZWQkKCGDRum1q9f79k+ffp0VVxcrJRS6tChQyojI0MlJiaqmTNnKqvVerZTnpO5c+eq5557rsW24uJiNX36dE8caWlpKi0tTY0YMUI9+uijXonjVD/5yU/UyJEj1ahRo9TVV1+tSkpKWsWlVPNd8eTkZJWQkNAtcSUmJqr4+Hg1evRoNXr0aM8Ihp56v05c/7CkZFX14hbV9NlhtWjRIpWYmKiUUqqpqUnNnDlTJSYmqoyMDHXo0CGvxXLChx9+qAA1atQoz/v0n//8Rz333HOez9lf//pXNWLECJWWlqbGjRunPv74Y6/H1d7P5dS43G63+sUvfqESEhLUyJEjW4yO8qb6+noVGRmpampqPNt64v2aNWuWio2NVQaDQZnNZvXCCy+oiooKNWnSJJWUlKQmT56sKisrlVJKffbZZ+rWW2/17Pviiy+qxMRElZiYqF566aWznUqe1BN9l6umkYbVOwn44TD8kvpjsVjkST3hy+RJPdF3uY83AqCXp/REHyIJWfRKuuAA/IYPQGeSZZtE3yHTb4peSR8dgj46pKfDEKJLSQtZ9EruuqbmRWaF6EMkIYteRylF/Zpd2D470tOhCNGlJCGLLteR2bL27NnDxRdfTGpqKmlpabz22muesltuuYWhQ4d6Zvras2dPi31Vgw2cbnQmuaEn+hZJyKLLLV26lMmTJ5Ofn8/kyZPbnCY0KCiIv//97+zdu5e8vDzuvvvuFlN7Pvnkk+zZs4c9e/a0miLVfbwJQG7oiT5HErLoch2ZLWvYsGGeiZ0GDBhATEwM5eXlHTq+q6Z5yJtMTC/6GknIost1ZLasU23fvh273U5iYqJn2+9+9zvS0tJYsGABNputRX13TSOav4EXXlmOxWLBYrF0OJkL4cvkST1xTs40Q9zcuXNbdD9ERES0u+pCaWkpEyZMYMWKFYwfP96zLTY2FrvdTnZ2NomJiSxevNizj/PbWlSdFWPiyUlf5Ek94eM69KSejEMW5+RMM8R1ZLYsgNraWq688koee+wxTzIGPK1rf39/fvrTn/LUU0+12M8QEwYxLae3FKIvkC4L0eU6MluW3W7nuuuu4+abb/asEnPCiZnvlFKsWbOmxQxlyu7EWVyNsju9eAVC9AxJyKLLLVy4kE2bNpGcnMzmzZtZuHAhADt27GD+/PkAvP7663zwwQcsX7681fC2OXPmMGrUKEaNGkVFRQUPPPCA59iuijoaN36Jq6Ku+y9MCC+TPmTRq9j2FmPbfpiQWePQBfp5tksfsvBxMtub6Hvc1Q1oAcYWyViIvkISsuhVXNWN6CJk/LHomyQhi15DKYW7pgF9RHBPhyKEV8iwN9GrBF85Gs2g7+kwhPAKScii19A0DX2kzIEs+i7pshC9hrOkGnv+ydWIhehrJCGLXsP+9TFse75B0zo0gkiIXkcSsug13NWNckNP9GmSkEWvoFxu3Meb0ElCFn2YJGTRK7hrGkEpaSGLPk0SsugV3CcmpY+ShCz6Lhn2JnoFY2IM+gEmtABjT4cihNdIQha9hsxfIfo66bIQPk8pReP7B3AWt73qiBB9hSRk4fPctU04D5fjbrT3dChCeJUkZOHz3JX1AOjlhp7o4yQhiy5XVVVFZmYmycnJZGZmtrvAqV6v96wWkpWV5dl+5MgRxo0bR1JSEjfeeCP2b2tBp6EzybSbom+ThCy63NKlS5k8eTL5+flMnjyZpUuXtlkvMDCQPXv2sGfPHtatW+fZ/utf/5oFCxZw8OBBIiIiKPoyH11EMJpOPq6ib5MlnESXS0lJYcuWLZ5VpydMmMBXX33Vql5ISAj19fUttiml6NevH8eOHcNgMLB161bs7x7gokk/JPDipHbPKUs4CR8nSziJnlFWVkZcXBwAsbGxlJWVtVnParVisVgYP348a9asAaCyshKTyYTB0DwiMz4+njtW/ZGA8Ymt9s/JycFisWCxWCgvL/fS1QjRfWQcsjgnU6ZM4dixY622P/bYYy1ea5rW7uxsR48exWw2c/jwYSZNmsSoUaMIDw9vs25bx8jOziY7OxtobiEL0dtJQhbnZPPmze2W9e/fn9LSUk+XRUxMTJv1zGYzAAkJCUyYMIHdu3dz/fXXU1NTg9PpxGAw4Nh1lGWz7/HKNQjha6TLQnS5rKwsVqxYAcCKFSu45pprWtWprq7GZrMBUFFRwccff8yIESPQNI2JEyfyxhtvAND0TQUD+sd1X/BC9CBJyKLLLVy4kE2bNpGcnMzmzZtZuHAhADt27GD+/PkA7N+/H4vFwujRo5k4cSILFy5kxIgRAPzhD3/g6aefJvWC4cQFmTCPSu6xaxGiO8koC+GznGW1NK7/nMBJIzAOjjpjXRllIXycjLIQvZurvBYAfb/QHo5EiO4hCVn4LF1IAMakGHRBMsub+H6QURbCZxmHRGMcEt3TYQjRbaSFLHySsjtRNkdPhyFEt5KELHyS40g5da9+irvO2tOhCNFtJCELn+Q8dhwt0IgW4t/ToQjRbaQPWfgcpRSuY8fRx4a3+9i1NzgcDoqKirBapVUuOkev12MymYiOjkZ3HrMSSkIWPkfVWVGNdgyxbc9r4S1FRUWEhoYyZMiQbv2PQPRuSikcDgdlZWUUFRUxaNCgcz6WdFkIn+MsrQFAH2vq1vNarVaioqIkGYtO0TQNPz8/zGYzDQ0N53UsScjC5xjMkQRckoQuPLDbzy3JWJyr8+mqOEG6LITP0YX445ciEwqJ7x9pIQuf4qpuwH6wDOV09XQofd7KlSuZOnXqOe2bmprKli1bujagbuLLsUtCFj7FcehbrB/lg1vmsTrVkCFDzjgH9bmYM2cOGzduPGu9W265hQceeKDFtr179zJhwoROna+goABN0wgJCfF8jR49ulPH6Kyuir27SJeF8CnO4mr0/cPQ/OSj2VfV1NR4lugSLUkLWfgMd50Vd1UDhvjIng6lV3n++edJSkoiMjKSrKwsSkpKPGUbN24kJSWF8PBwfvGLX/CjH/2IF154AYDly5dz6aWXAs1DtxYsWEBMTAxhYWGMGjWKL7/8kpycHFauXMkTTzxBSEgIV199NdCyxe5yuXj88cdJTEwkNDSUsWPHUlhY2KlreOihh/jJT37ieX2iNe10OgGYMGECixYt4gc/+AGhoaFMnTqViooKT/2PPvqISy65BJPJxMCBA1m+fHmHYrfZbNx9990MGDCAAQMGcPfdd3sWTtiyZQvx8fH88Y9/JCYmhri4OF5++eVOXVdnyX9Twmc4Cpr/gfnKhEI1d7+NY0/rdQO7kjE9FtMz0895/3fffZff/OY3bNy4kdTUVP7nf/6HWbNm8cEHH1BRUcHMmTNZvnw5WVlZPPvsszz//PPcdNNNrY6zceNGPvjgA77++mvCw8M5cOAAJpOJ7OxsPvnkE+Lj43n00UfbjOHpp59m1apVrF+/nmHDhvHFF18QFBR0ztfUnldffZW3336bgQMHMn36dJ566imWLl3K0aNHmT59Ojk5OcycOZPa2loKCwtJT08/a+yPPfYYn376KXv27EHTNK655hoeffRRHnnkEQCOHTvG8ePHKS4uZtOmTcycOZNrr72WiIiILr8+kBay8IKqqioyMzNJTk4mMzOT6urqVnXee+890tPTPV8BAQEUfHEAXVQI8375c4YOHeop27NnTw9cRe+wcuVK5s2bx5gxY/D392fJkiVs3bqVgoIC1q9fT2pqKjNmzMBgMHDXXXcRGxvb5nGMRiN1dXUcOHAApRTDhw/3rBx+Ni+88AKPPvooKSkpaJrG6NGjiYpqf0GB6OhoTCYTJpOJp556qsPX+tOf/pRhw4YRGBjIDTfc4PlcvPrqq0yZMoXZs2djNBqJiooiPT29Q8dcuXIlixcvJiYmhn79+vHggw/yyiuveMqNRiOLFy/GaDRyxRVXEBISwldffdXhmDtLWsiiyy1dupTJkyezcOFCli5dytKlS/nDH/7Qos7EiRM9/6CqqqpISkpi4HWXEGj0gzfhySefZObMmT0Rvsf5tFy7S0lJCWPGjPG8DgkJISoqiuLiYkpKShg4cKCnTNM04uPj2zzOpEmTuPPOO7njjjs4evQoM2bM4KmnniIsLOysMRQWFpKYmNjhmCsqKs6pD/nU/0yCgoKor68/p/OfqqSkhMGDB3teDx48uEWXT1RUVItYTz2vN0gLWXS5tWvXMnfuXADmzp3LmjVrzlj/jTfeYPr06QQFBaEZpY3QGQMGDODo0aOe1w0NDVRWVmI2m4mLi6OoqMhTppRq8fp0d911Fzt37mTfvn18/fXXPPnkk8DZH5YZOHAghw4dOq/rCA4OprGx0fP62LGOdxWd6fxni/309++bb75hwIABHT53V5OELLpcWVmZ59fd2NhYysrKzlg/NzeXpZfPx7bnG8+23/3ud6SlpbFgwQLPTZbT5eTkYLFYsFgslJeXd90F+CiHw4HVavV8OZ1OZs+ezcsvv8yePXuw2Wz89re/Zdy4cQwZMoQrr7yS//73v6xZswan08mzzz7bbqL77LPP2LZtGw6Hg+DgYAICAjxPnvXv35/Dhw+3G9f8+fNZtGgR+fn5KKX44osvqKys7NS1paen88EHH/DNN99w/PhxlixZ0uF958yZw+bNm3n99ddxOp1UVlZ6fvs6W+yzZ8/m0Ucfpby8nIqKCn7/+9+3uLnY3SQhi3MyZcoURo4c2epr7dq1LeppmnbGVkppaSmuynrCnQY0/+bW8ZIlSzhw4ACfffYZVVVVrbo7TsjOzmbHjh3s2LGDfv36dd3F+agrrriCwMBAz9dDDz3ElClTeOSRR7j++uuJi4vj0KFD5ObmAs19tf/85z+5//77iYqKYt++fVgsFvz9W09pWltby89+9jMiIiIYPHgwUVFR3HfffQDceuut7Nu3D5PJxLXXXttq33vuuYcbbriBqVOnEhYWxq233kpTU1Onri0zM5Mbb7yRtLQ0xo4dy1VXXdXhfQcNGsT69ev54x//SGRkJOnp6Xz++ecdiv2BBx7AYrGQlpbGqFGjGDNmTKtxy91JVp0WXS4lJYUtW7YQFxdHaWkpEyZMaPdGyJ///GeSGoL4oXk4obPGofkbW5Rv2bKFp556irfeeuuM5+yKVaf379/P8OHDz+sYvsztdhMfH8/KlSuZOHFiT4fTJ53hMySrTouekZWVxYoVKwBYsWIF11xzTbt13/znG1xsTsEwKMqTjEtLS4HmPs81a9YwcuRI7wfdR23YsIGamhpsNhuPP/44SinGjx/f02GJdkhCFl1u4cKFbNq0ieTkZDZv3szChQsB2LFjB/Pnz/fUKygo4MKooRjcGn7DT95ImTNnDqNGjWLUqFFUVFT06K+Qvd3WrVtJTEwkOjqaf//736xZs4bAwO6fRU90jHRZiB7lrmvCcaQCv1Hx5zX1pXRZCF9wvl0WMsZI9ChdaCD+aQPPXlGI7wHpshA9QimF9dNDuCrqejoUIXyGJGTRI5zfVGHfX4Kr6vyWvBGiL5GELLqdcitsuwrQhQViTOrf0+EI4TMkIYtuZ99fgrumEX/LEDSdrGEnxAmSkEW3cjfasO06iiE+AsOg9mcEE93jf//3f5kyZUpPh9Ept9xyC0888URPh+EVkpBFt9ICjPiNNBMwPlFWeO6gU5c80ul0BAYGel6vXLmyW2OJjY0lKCioRUydnbeiM9r6D2P58uXcf//9XjtnT5Jhb6LbKJcbTa8j4MLBZ68sPE6d7nHIkCG88MILPdqq3bhxo2elEdG1pIUsuoXjm0rq/7UDV5X35pL9vvr4448ZN24cJpOJAQMGsGDBAs/SR1arFU3TyMnJITExkYiICBYsWNBif6UUd911FyaTicTExHNaTDUvL4+kpKQW22JjY/noo4+A5qc358yZw+zZswkNDSUtLa3FwgMFBQVcc801REdHEx0dzb333svu3bu5++672bJlCyEhIZ75kGfNmtViBZBnn32WxMREoqKimDFjhmd2wY5cu6+RhCy8zlleS9P7B9ACjehCe9djuw1vf9Hqy76/eQJz5XS1XZ7fnBDcVkeb5Y7DzVOFuuvbnla0s4xGI8uWLaOyspIPP/yQf//73551807Iy8tj9+7d7Nq1i5dffpktW7Z4yj744AMsFguVlZXceeedLR5v70qrV69m3rx51NTUMHnyZO6++26geVrR6dOnM3z4cL755hsKCwu5/vrrufDCC3nmmWeYMGEC9fX1bU4dun79eh555BFWr15NcXEx0dHRrZaoOtO1+xpJyMKrnKU1NOZ9iRboR9DkVDSjvqdD6nMuuugiMjIy0Ov1JCYmMn/+fN5///0WdX77298SFhbG0KFDueyyy1q0TlNSUrj55pvR6/XMnTuXo0ePUlNT0+75pk+f7lmCadasWR2Oc9KkSWRmZqLX67nppps8MXz00UfU1tby+OOPExQURGBgIJdcckmHjrly5Uqys7NJS0sjICCAJ554gs2bN7dI3me6dl8jfcjCa5xlx2nc9CW60ECCpo1EF+TX0yF1WvD0tHbLNIP+jOW6AOOZy0Naz0t8Lvbt28e9997Lrl27aGpqwul08oMf/KBFnfaWP2qrDJr7rU0mU5vne/vtt8+pD/lMSzANHTrUMyF+Z5SUlDBp0iTPa5PJRFhYGMXFxZ74z3TtvkZayKLLnZiwSh8Vil9KHEHT09AFdU3yEa397Gc/Y8yYMRw6dIja2loWL15MJycNO2+nL8HkcDioqqrq0L4DBw6koKAAt9vdqqyzSzDV1NRQW1uL2WzuYOS+RRKy6DLK6cZ+oISGdbtRDieaQUfAuER0Acaz7yzOWV1dHeHh4YSEhLB3716ef/75bo9h+PDhVFVV8c477+BwOHjwwQfbTLBtufTSSwkNDWXRokU0NjbS1NTEJ598AjQvwVRYWIjD4Whz39mzZ/P888/z5ZdfYrVaWbhwIZMmTWp3dW1fJwlZnDd3gw3b599Q/6/PsG49hGbUo2zOng7re+NPf/oTL7zwAiEhIdxxxx3ceOON3R5DdHQ0f/7zn5kzZw7x8fHExsYSHR3doX2NRiPr16/n888/Jz4+nkGDBrF69WoALr/8coYMGUJMTEybK2ZfddVV/OY3vyErK4sBAwZw7NgxXnnllS69tu4k8yGLTlNuBU4Xmp8Bd20T9f9qnodYHxuO/+hBrP5oEw89/BD79+9n+/btWCyWNo+Tl5fHr371K1wuF/Pnz/dMZH/kyBFmzZpFZWUlY8eO5ZVXXsHP78z9zzIfsvAFsoST8IpT/6N2FFZh21tM0yf5NLy1h7p/fELT1oMA6MIC8R+XQMj1FoKnp2EYYGLkqJG8+eabXHbZZe0e3+Vycccdd/D222+zb98+Vq1axb59+wD49a9/zYIFCzh48CARERG8+OKL3r1YIXxEt4yyUE43zsLvHq88pY2tiwxGbwpCOVwny0+hiwpBHx6EsjlwFlWfvjuGmFB0oYG4m+w4i6tb7W+IDUcXEoC7wYaz5MQwnpNHMJgj0AX5466z4ixtPczHMDASXaAfruONuI4db1VuHBKN5m/EVdWA69va1uWJ/dCMBlwVdSfn/T3lAozJsWgGHc6yWtyVre/8GofHoWkaztIaXNUNLfbXNA2/Ec3LHjmLq3HVNLbYV9Pr8LsgDgBHQQXumkaU2w0uBS43+Bs8T8xZdxXgKq9D2ZwouxNlc6IPDyT4qnQAbLsKcFc1gJ8BfWQwfimx6GPDPefyH9HyBkpHWpnbt28nKSmJhIQEoHmw/9q1axk+fDjvvvsur776KgBz587loYce4vbbbz/rMYXo7bopITtp2nKg1Xb/sUOaE7LVTtP7rVclDhifiD48CHeDjaYP2ij/4TD8QgNx1zZh/fDrVuWBk4Y3J+TqBqwftS5vHorlj6uiDuvH+a3LrxzdnJDLarF+crBVuT4mDL2/EWdpDbbth1uVG+Ij0IwGnEXV2HYfbVVuTOgHBh3Owkrs/y1qXX5BHGjNCdVxoPS0k59MyI5D3+I49G2LYs3fcDIhH/oW5zff/Yen14Fea35A47uErJoc4HA133wLD0TzM6ALO/kAR9Ck4WA0oPkbumz+ieLiYgYOPLlSSHx8PNu2baOyshKTyYTBYPBsLy4ubvMYOTk55OTkAFBeXt4lcQnRk7olIWt+RoKvG3PqluY/v7v7rgX5EzxjbKv9Ttyd14UHETzjZD/kiZxwYn99VAghM1v3U2oBzf2O+thwQmZmnH56z/6G+EhCfpzB6RW0wOZy49B+GMwRrY//XbnfsFiMQ/u1cf7vylPNGFPauOvr1/z2+48ehN/I1jcsTsQZYBlKwJj2538IuDiJgHGJrfY7IXDCBc3bNK3NhBr4g+R2jw20+XTdlClT2nxy6rHHHjvjKtNdKTs7m+zsbIB2+6mF6E26JyHrNPSm4PbL9Tr04UFnKW//kVvNoEc7wyO5zeXtPyGmGfVnfILMF8rhbOXt0/Rdf6vgXOY7OJXZbKawsNDzuqioCLPZTFRUFDU1NTidTgwGg2d7d1FKySx04px0dJjfmchNPdEjMjIyyM/P58iRI9jtdnJzc8nKykLTNCZOnMgbb7wBwIoVK7qtxR0QEEBlZWW3P1QhejelFHa7neLiYoKD2294doQMexNdbvXq1fzyl7+kvLwck8lEeno6GzZsoKSkhPnz57N+/XqgeWKYu+++G5fLxbx58/jd734HwOHDh5k1axZVVVVceOGF/OMf/8Df/8xP+nXFsDeHw0FRURFWq/W8jiO+fwwGA+Hh4URHR7f3CHiHfu2ShCz6hK5IyEJ4kYxDFkKI3kQSshBC+AhJyEII4SMkIQshhI/o7E09IXySpml5SqnLezoOIc6HJGQhhPAR0mUhhBA+QhKyEEL4CEnIQgjhIyQhCyGEj5CELIQQPkISshBC+AhJyEII4SMkIQshhI+QhCyEED7i/wM2ygHZ5mq36wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\r\n",
    "import matplotlib.pyplot as plt\r\n",
    "\r\n",
    "# Logistic函数\r\n",
    "def logistic(z):\r\n",
    "    return 1.0 / (1.0 + paddle.exp(-z))\r\n",
    "\r\n",
    "# Tanh函数\r\n",
    "def tanh(z):\r\n",
    "    return (paddle.exp(z) - paddle.exp(-z)) / (paddle.exp(z) + paddle.exp(-z))\r\n",
    "\r\n",
    "# 在[-10,10]的范围内生成10000个输入值，用于绘制函数曲线\r\n",
    "z = paddle.linspace(-10, 10, 10000)\r\n",
    "\r\n",
    "plt.figure()\r\n",
    "plt.plot(z.tolist(), logistic(z).tolist(), color='#e4007f', label=\"Logistic Function\")\r\n",
    "plt.plot(z.tolist(), tanh(z).tolist(), color='#f19ec2', linestyle ='--', label=\"Tanh Function\")\r\n",
    "\r\n",
    "ax = plt.gca() # 获取轴，默认有4个\r\n",
    "# 隐藏两个轴，通过把颜色设置成none\r\n",
    "ax.spines['top'].set_color('none')\r\n",
    "ax.spines['right'].set_color('none')\r\n",
    "# 调整坐标轴位置   \r\n",
    "ax.spines['left'].set_position(('data',0))\r\n",
    "ax.spines['bottom'].set_position(('data',0))\r\n",
    "plt.legend(loc='lower right', fontsize='large')\r\n",
    "\r\n",
    "plt.savefig('fw-logistic-tanh.pdf')\r\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**********\n",
    "\n",
    "**说明**\n",
    "\n",
    "在飞桨中，可以通过调用`paddle.nn.functional.sigmoid`和`paddle.nn.functional.tanh`实现对张量的Logistic和Tanh计算。\n",
    "\n",
    "**********"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.1.2.2 ReLU型函数\n",
    "\n",
    "常见的ReLU函数有ReLU和带泄露的ReLU（Leaky ReLU），数学表达式分别为：\n",
    "\n",
    "$$\n",
    "\\mathrm{ReLU}(z) = \\max(0,z),（4.6）\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\mathrm{LeakyReLU}(z) = \\max(0,z)+\\lambda \\min(0,z),（4.7）\n",
    "$$\n",
    "\n",
    "其中$\\lambda$为超参数。\n",
    "\n",
    "可视化ReLU和带泄露的ReLU的函数的代码实现和可视化如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADuCAYAAAAOR30qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XlcVXX+P/DXuSv3ci8gKKu4sKiAIgKKmZqaaJqaC+6OuGRONr8Zq7GaKasxG82prG9NzTjlXpiauW+p4VIq4p77hgoCArLcfTuf3x/oTQKRq5d7Ltz38/HwMcM959zzOofbm8/9nHM+H44xBkIIIcITCR2AEEJIJSrIhBDiJqggE0KIm6CCTAghboIKMiGEuAkqyIQQ4iaoIBNCiJuggkwIIW6CCjIhhLgJiYPr02N9xC0988wz2L59u9AxCHkQri4rUQuZNArFxcVCRyDksVFBJoQQN0EFmRBC3AQVZEIIcRNUkAkhxE1wDo6HXOPKFosFubm5MBqNzklFiIPy8/MREhICsVgMPz8/NG3aFCIRtTeI26jTXRaO3vZWo9zcXKjVarRq1QocV6f9EgIAyMnJQVlZGaRSKeLi4gAAVqsVV69ehclkglwuR0REBCSS2j+qjDG0a9cOFosFhYWFyM3NRYsWLVxxCIQ4jVOaEEajEQEBAVSMicMCAgIQHR1d5bWCggKo1Wp06NABarUaBQUFdXovjuMgk8kQFhYGnU5XH3GJB2KMwZavccm+nPadjooxeRRqtbpa67esrAwBAQEAKgt2WVmZQ+9JXRXEmTRz9uJ2hy9gveHY5/BROKXLghBnslgskMlkAACpVAqLxVLjekVFRSgqKgJQ2c1BiLNpVx6DZsEBKEfGQRzuW+/7o6aEh4mLi0NmZqbQMeqstm9ezZo1Q2xsLGJjYx/ax0yIowy7L8Km10D5507wWzTYJb0Ajb4gt2rVCgqFAiqVCsHBwZg0aRK0Wm2dts3MzETz5s1rXNarVy989dVXdV4/JycHHMdBpVLZ/3Xs2NGxg3HQpEmT8NZbb1V57cyZM+jVq1e97vdxSaVSmM1mAIDZbIZUKhU4EfE05nOFMJ/NBdOYof5zD3Ay1/zBb/QFGQA2bdoErVaLEydO4Pjx45g3b55gWcrKyqDVaqHVanHy5EnBcrgzX19flJSUAABKSkrg5+cncCLiSWzFWug3nwJEHBRPx0IS4uOyfXtEQb4nODgY/fv3x4kTJ+yvmUwm/PWvf0WLFi0QFBSEP/7xjzAYDC7N9e6772LChAn2n++1pu/1i/bq1QuzZ8/Gk08+CbVajX79+lUZTOfAgQPo1q0b/Pz8EB4ejqVLl2LRokX45ptvsGDBAqhUKgwePBhA5TeGXbt2Aag89pkzZyI0NBShoaGYOXMmTCYTgN9a+x999BECAwMREhKCJUuWOP3Yr169ivPnz8NoNOLkyZMoKipCSEgIKioqcPr0aVRUVCA4ONjp+yWkJsxqQ8X/7QPXRAZZq2DIE8Jcuv96aYeXzdwGy4m63ar0qKQJwfD7ZIBD2+Tm5mLbtm3o06eP/bU33ngDV65cwYkTJyCVSjFu3DjMmTNH0FZ0Tb799lts27YN4eHhGDBgAD788EPMnz8f169fx4ABA7Bo0SKkpaWhoqICN2/eREJCAn755Rc0b94cc+fOrfE933//fRw6dAgnTpwAx3F47rnnMHfuXLz33nsAKm8/Ky8vR15eHn788UekpaVh6NChaNKkidOOKyIiosbX27Zt67R9EFJX5TO3w/DNSfj862kono1x+f49ooU8dOhQqNVqhIeHIzAwEP/4xz8AVN5fuGjRIixcuBD+/v5Qq9X4+9//jlWrVtVblqZNm8LPzw9+fn748MMP67zd5MmT0aZNGygUCowaNcreyv/222/Rt29fjB07FlKpFAEBAUhISKjTe37zzTd4++23ERgYiGbNmuGdd97BihUr7MulUinefvttSKVSDBw4ECqVChcuXHDsgAlpIDT/PgjdF0fg/XwyVM+nCJKhXlrIjrZc69v69evRt29f7N27F+PGjUNxcTH8/PxQVFQEvV6PpKQk+7qMMdhstoe+p0QiqXY7lsVieegFqOLi4ke6I+D+r+1KpdJ+YfLmzZuIjIx0+P0A4NatW2jZsqX955YtW+LWrVv2nwMCAqpkvX+/hDQm+vW/gveyQPlqMnzm9xUsh0e0kO956qmnMGnSJPz1r38FUNlaVSgUOHPmDMrKylBWVoby8vI6FZ0WLVogJyenymvXrl2rUuDqytvbG3q93v5zXZ9MA4Dw8HBcuXKlxmUPu00nNDQU169ft/9848YNhIaG1nnfhDQGpqwbsOQXgZUY4fNmH3Bi4cqiRxVkAJg5cyZ+/PFHnDx5EiKRCNOmTcPLL7+M27dvAwDy8vKwY8eOKtsYjcYq/xhjGD16NJYsWYKsrCwwxnDx4kUsXLgQY8aMcThTQkIC9u3bhxs3bqC8vNyh/uvx48dj165dWL16NaxWK0pKSuzdGUFBQbh69eoDtx07dizmzp2LoqIiFBcXY86cOVUuLhLS2FlulMLw80UwMw/v4Z0g9lMKmsfjCnKzZs0wceJEzJkzBwDwwQcfICoqCl27doWPjw/69u1bpZ80Ly8PCoWiyr8rV66gf//+mD9/PiZPngxfX18MHDgQ6enpeOGFFxzOlJqaitGjRyM+Ph5JSUkYNGhQnbdt0aIFtm7dio8++gj+/v5ISEiw3043depUnD17Fn5+fhg6dGi1bd966y0kJycjPj4eHTp0QGJiYrX7lglprHi9Gdrlh8EpJPBKjIA0qpnQkZwz/Oa5c+cQE+P6K5KE3HP27FnExsbaf6bPJKkN43mUjvsepiM34TOvD7xH1e1C+GOgSU4JIaQmmnl7YfjuDFTTu7iiGNcZFWRCiEfRLs8GQiVQzuoC1awnhY5TBY3IQgjxGIZdF2Ez6sBKTPD9R1+3GzaYCjIhxCOYzxTAfC4XzGyDenJXiBQyoSNVQ10WhJBGz3ZHB/3W04AIUKbGQRzsugGDHEEFmRDSqDGrDaVjvodpw1XIIkIgi3ffh5+oIBNCGi2e51E+awdMP16FakpnKAa4962QVJAJIY2W7r8HwcV4QzW7O7ynJAod56GoID+CpUuXonv37kLHaPD2799Pw2ySeqNfdxq8zAZWbIL6nd5Cx6mTRl+Q7x+QXWiZmZkQiURQqVRQq9Vo27atQ4O+1zQlE1B9QPuHrQ9UDoovlUqrTCm1YMECxw7IQRzH4fLly/afe/ToQcN5knphOnQdlsJisGIj1C/1gEgsFjpSndBtby4WGhqK3NxcMMawbds2DBkyBN26dROkpTh69GisXLnS5fslpD5Zb5TCcPASwHHwHpkEsa9C6Eh11uhbyLXZvHkzEhIS4Ofnh27duuHUqVP2ZfPnz0dkZCTUajViY2Pxww8/PPB9Zs2ahe7du6O8vBz+/v44ffq0fdnt27ehVCrt09Xfw3EcBg4cCH9//yr7PX/+PFJTU+Hv74+2bdti9erVTjziuvn9t4r7p5i61xpftmwZWrRogaZNm+L999+3r2uz2fDPf/7Tfu6SkpJw8+ZN9OzZEwDQsWNHqFQqfPfdd9UmhT137hx69eoFPz8/xMXFYePGjfZlkyZNwksvvYRnn30WarUaKSkpDxx2lHguZrDgzsi1sB4uhFdyBKQRAUJHcki9tZB1205Ve03aqilkMaFgVhv0P56pvjwqCLLoIPBGCww/nau2XNY2BNKIZuC1JohU8sfKd/z4cUyZMgWbNm1CcnIyVq5ciSFDhuDChQuQy+WIjIzE/v37ERwcjDVr1mDChAm4fPkyQkJC7O/B8zymT5+OGzduYOfOnVAqlRgzZgxWrlyJDz74AACQkZGBp59+Gs2aVR1Jiud5bN68GcXFxYiKiqo8ZzodUlNTMWfOHGzbtg2nT59Gamoq2rdvX2XgHHdw4MABXLhwARcvXkSXLl0wfPhwxMTE4OOPP0ZGRga2bt2KNm3a4NSpU1Aqldi3bx84jsPJkyftx5uZmWl/P4vFgsGDB2PKlCnYuXMnDhw4gOeeew7Z2dn2bw+rVq3Ctm3bkJiYiPT0dLz55pv1OrsLaVh4mw1l0zbAciQP/n8fA6/urYWO5DCPbSEvWrQI06dPR0pKCsRiMdLT0yGXy3Ho0CEAwMiRIxEaGgqRSITRo0cjOjoaWVlZ9u0tFgvGjh2LO3fuYNOmTVAqK8dRTU9PR0ZGBu6NordixQr84Q9/sG9369Yt+Pn5QaFQYNiwYfj444/RqVMnAJUt9latWmHy5MmQSCTo1KkTRowYgTVr1tTLOVi9erV9Oik/P78qs4U8zDvvvAOFQoGOHTuiY8eO9iE/v/rqK8ydOxdt27YFx3Ho2LEjAgIe3ko5dOgQtFot3njjDchkMvTp0weDBg1CRkaGfZ1hw4ahS5cukEgkGD9+fJXJagnRfrofoiQ/qP/VF4rn2gkd55HUWwvZe0D8A5dxEnGty0Ve0tqXP2brGACuX7+OZcuW4bPPPrO/Zjab7UVp+fLl+Pjjj+2zgmi12iozPV++fBknT55EVlYWZLLfHsFMSUmBUqlEZmYmQkJCcPnyZQwZMsS+/F4fsslkwhtvvIE9e/Zg5syZ9kyHDx+uMu291WqtUtBrcm+aJYvFUmXKpYdNKTVq1KhH7kN29pRSt27dQnh4OESi39oILVu2RF5e3kP3SYh26RGgiRgotEL1Wjeh4zwyj20hh4eH480337RP3VRWVga9Xo+xY8fi+vXrmDZtGj7//HOUlJSgrKwM7du3x/1jR8fExGDJkiUYMGBAtTsF0tPTsXLlSqxYsQJpaWnw8vKqtn+5XI4PPvgAp0+fxvr16+2ZnnrqqSqZtFotvvzyy1qPJSQkBFKp1O2nlKpNaGgobt68CZ7n7a/duHEDYWGunYadNDyGnRdgM+vBFxjgM/OpKn/UG5qGm9wBFoulyhRMVqsV06ZNw3/+8x8cPnwYjDHodDps2bIFGo0GOp0OHMfZ+32XLFmCX3/9tdr7jh07Fv/85z/Rt2/fKkVowoQJ+OGHH7By5UpMnDjxgblkMhleffVV++wlgwYNwsWLF7FixQpYLBZYLBYcOXIE58791p9us9mqHIvZbIZYLMaIESPw5ptvoqSkBBaLBRkZGTh79iwGDHB8wtmEhASsWrUKFosF2dnZWLt2bZ23ff755zF79mxcunQJjDGcOnUKJSUlAGqfUureN4sFCxbAYrEgMzMTmzZteqQpsYjnMP9aAPPFPLAKM9RTnoDIq/ZJht2dRxTkgQMHVpmC6d1330VycjL+97//4U9/+hOaNGmCqKgoLF26FAAQGxuLV199FU888QSCgoJw+vRpPPlkzeOmpqen4+2330afPn3sLdTw8HAkJiaC4zj06NGj1mxTpkzBjRs3sGnTJqjVauzcuROrVq1CaGgogoOD8frrr8NkMtnXnz9/fpVj6dOnDwDgiy++gL+/P+Lj4xEYGIjPP/8cW7ZsQVBQkMPn67333sOVK1fQpEkTvPPOOxg3blydt33llVcwatQo9OvXDz4+Ppg6dSoMBgOAyrs10tPT4efnV+3uEZlMhk2bNmHbtm1o2rQpZsyYgeXLl6Ndu4bZF0jqH19mQOn4dbCeL4Oyf3uIg9RCR3psNIVTPZkyZQpCQ0Mxd+5coaN4BJrCybPwJgvuDMmA6accNN01EfKerYSO9DB1GniZHgypBzk5OVi3bh2OHz8udBRCGh2e51GxcC/EXfzhO6Z9QyjGdeYRXRauNHv2bLRv3x6zZs1C69YN7z5IQtyd7j8HIQqSQ+TnDdVk9x8wyBHUQnay9957D++9957QMRq8wsJCFBUVgeM4KBQKtGrVqkFfPSfOoVt7CrzcBpZvhM/rTwsdx+noE07cjtlsRmFhIWJjYxEXFwfGGO7cuSN0LCIw4y85sBbfqRww6E8NZ8AgRzitIDt4cZCQh+J5Howx8Dxf6wMuNW1HGhdbvgblf9wC/qYW3qOSIfZpOAMGOcIpXRZeXl4oKSlBQECA283iShoemUyGoKAgnDp1CiKRCD4+PvD19a22XlFRkX3QJqvVCsYYLBYLCgsL4e3t7erYpJ7YNEaUDMmA7WoZvEcOh7S1v9CR6o1TbnuzWCzIzc2F0Wh0Tiri0XieR1FREZo2bQqRSISioiIolUqoVKoHbpOfn4+QkBBIJBL4+vratyUNG2+zoeKD3bBdK4NicDyUQxrsfel1aqk6pSAT4kxr1qzB9u3b8fXXXwOoHFfk0KFD+OKLLx64TXJyMrKzs10VkbhI+Uc/gfOXAHoOPi816Fl66lSQqQlB3E6LFi1w6NAh6PV6MMawe/duesjDA2mWZIHzl4AvNEH1YsMdMMgRVJCJ20lJSUFaWhoSExPRoUMH8DyPF154QehYxIUM28+DtxrA5xvg83LDHjDIEdRlQRoF6rJoPKyXSlAyfjVkQ1rD54UnIQ5s+GNUgB6dJoQ0NLY7epQM+hZ8iR6qMZ0bSzGuM8/4HkAIcXu8yQLNf/aDi1TB/4cxkEY1rPnwnIEKMiFEcPcGDBKFKKEc1R7yHo5PrNAYUEEmhAhO+8UvEAXJgVIbVJM6Cx1HMFSQCSGC0q0+AabgwecboPpL7RM6NHZUkAkhgjEfz4fuq2NghQao/9SzUQ4Y5Ai6y4IQIghrbjlKBn8LTsRBtSwNYp/qkwF7GmohE0JczlZugHbpQYjClAjYPA6SEB+hI7kFKsiEEJfibTZo/r0fXIgS6te7QxofLHQkt0EFmRDiUpqF+yAKUUBkFkM5vIPQcdwKFWRCiMtoFh8GFyABKzTBe/oTQsdxO1SQCSEuYcq8BlPmVfD5Bqg9aMAgR9BdFoSQeme5WIyS4d9BHKxCk4VDIJLXfUouT0J/oggh9cpWUAHtisMQhXgjYPM4iANoeq0HoYJMCKk3vMEMzZJDEIWr4Ptpf0giGu98eM5AXRaEkHrB8zwqPt0HUbACIrECir5thI7k9qiFTAipF9p///zbgEETk4WO0yBQQSaEOJ1+3VlYb5VWDhg0s6fQcRoM6rIghDiV+egtlE1YB0mnEDTdPoFub3MAnSlCiNNYLhdV3lER7oOAdaMhUsuFjtSgUAuZEOIUtjI9dOuOQ9zWD03GJUEcpBI6UoNDBZkQ8th4iw2aLw6AC/KCNMAf8i4thI7UIFGXBSHksWk+2Vs5YJBFAuXQ9kLHabCoIBNCHov2q2wwzgZ22wz1H7sJHadBo4JMCHlkxt1XUf7iVlj334b65aeEjtPgUUEmhDwS04k8aJcehqRtAPyXj4BIRpekHhcVZOKWysrKkJaWhnbt2iEmJgYHDx4UOhK5jzW/AobdZyHtEoQma0ZC5Evz4TkD/Ukjbukvf/kLnnnmGaxduxZmsxl6vV7oSOQuXm+GdukhcAFyyKJCIYsJFDpSo0EFmbid8vJy7Nu3D0uXLgUAyGQyyGQyYUMRAPcGDNpbOWCQVAlFn2ihIzUq1GVB3M61a9fQrFkzTJ48GZ06dcLzzz8PnU4ndCwCQPvhz+CaSIFyHqoJSULHaXSoIBO3Y7VacezYMbz44os4fvw4vL29MX/+/GrrLVq0CMnJyUhOTkZRUZEAST2L4fuz0Ly+G7YjpVD9uYfQcRoljjHmyPoOrUzIoygoKEDXrl2Rk5MDANi/fz/mz5+PLVu2PHCb5ORkZGdnuyih5zHuu4qKOXvA6YGme9LBedEUTA7i6rIStZCJ2wkODkZ4eDguXLgAANi9ezdiY2MFTuW5zJeKYDx+DbJnW6HJ2pFUjOsRXdQjbumzzz7D+PHjYTabERERgSVLlggdySPZyvTQ/3AcnFoKr+QoSEJ9hY7UqFFBJm4pISGBuiAEVm3AoM7hQkdq9KjLghBSI83cPZUDBlmlNGCQi1BBJoRUo1uUDe2cn8Gf10E9/Qmh43gMKsiEkCr0m89Cs/AA5AOi4Pt+qtBxPAr1IRNC7EzHcmG5UQiviTFQz+gJTiIWOpJHoYJMCAEAWPPKYcg8D0g4KId1gthXIXQkj0MFmRBSOWDQ8sPg/GWQt2kOWTsaMEgI1IdMiIdjjKH8vV0QBSsgVvnAq3eU0JE8FhVkQjyc9p/7of8gCyjioRqfKHQcj0YFmRAPpv32ODQLf4FiXAeoZ/UUOo7Ho4JMiIcy/nQZNk05FDPi0eSrIeC4Oo1/Q+oRXdQjxAOZL96G8fR1wMqgfqEbDRjkJqiFTIiHsd3RQ7/+BDiJCIqebSFp7id0JHIXFWRCPAiz8aj48Cdw/nJIwwMhT6YBg9wJFWRCPEj5qzugX3gMYl4O5eA4oeOQ36GCTIiH0Pz3EHRfHoH39GSoXugqdBxSAyrIhHgA/cYz4MUmKF9Jgu9H/YWOQx6A7rIgpJEzZd+EJfc2mNYCn1m9wYmpHeauqCAT0ohZc8tg2HehcsCgoQkQ+yuFjkRqQX8qCWmkmNECzf8OgvOWwCu+FWRtaMAgd0cFmZBGiDGG0qkbYfjiJCQ+vvDqFSl0JFIHVJAJaYQ0/9oHQ8ZpqGd2g/fYTkLHIXVEBZmQRkb7zTGgmQjK11Og+nsPoeMQB9BFPUIaEeNPl2DTacBKTPCd/TQNGNTAUEEmpJEwnyuE6fQNMAsP1cQUiJQyoSMRB1GXBSGNgK3cAP2Gk4BYBEWfGEjCfIWORB4BFWRCGjhmtaF07PcwLj8HacsgyDs1FzoSeURUkAlp4Mrf/BGmbZehnvkklINihY5DHgMVZOK2bDYbOnXqhEGDBgkdxW1p/nsQonbeUM1+Et4vJAsdhzwmKsjEbX366aeIiYkROobb0q//FbzEAj7fAPXsPkLHIU5ABZm4pdzcXGzZsgXPP/+80FHckinrBiz5RWDFRqhndIdIKhY6EnECKsjELc2cORMLFiyASEQf0d+z3iqH4eeLYGYeyuGdIPajAYMaC/q0E7ezefNmBAYGIikpqdb1Fi1ahOTkZCQnJ6OoqMhF6YTFjBaUpq2Bedt1eHVqDVl0M6EjESfiGGOOrO/QyoQ8ir/97W9YsWIFJBIJjEYjKioqMHz4cKxcufKB2yQnJyM7O9uFKV2P53mUTdsAw+KT8P9+FBTD6Y6KBqROj0xSQSZuLTMzEx9++CE2b95c63qeUJArFu4F8+LBaSXwmUVjVDQwdSrI1GVBSAOgXZEN+InASq1Qvfqk0HFIPaEWMmkUGnML2bD7IsxX8sFKTPCZ2QsiBY1R0QDVqYVMgwsR4sbM527DfDYXzGyDalJXKsaNHHVZEOKm+HIjSketgSX7NhRPx0IS4iN0JFLPqCAT4oZ4sxV3JnwP6/kSqNO7QJ4QJnQk4gLUZUGIm2GMQbNwLyQ9msHrubaQ94kQOhJxEWohE+JmtP89CC5QBo6TQPU8DRjkSaggE+JG9D+cBi+1Vg4YNPMpoeMQF6MuC0LchCnrBiwFxWAVFqhf6kEDBnkgaiET4gZsBRqUpm+A7VI5vEd0gthXIXQkIgAqyIQIjNeaUDJ8FfgbFVCN7wJpFA0Y5Kmoy4IQAfE2Gyr+by8k3QOh+uuTkCWFCh2JCIgKMiEC0n66H6IQBURqBZQ0epvHoy4LQgSiXX4EaCIGX2CC6iUaMIhQC5kQQRh+vACbUQ9WbILPy71oZhQCgFrIhLic9eodlL/yI/hcLdRTaMAg8htqIRPiQrYSHUoGfQu+QAv1H7pCHEwDBpHfUAuZEBfhTRZo/vszxIn+CFg3GpLoAKEjETdDLWRCXIDneWgW7oUoRAF5SBTkvVoLHYm4IWohE+ICuv8cBBckByuxQj25i9BxiJuigkxIPdN9fwq83FY5YNDLPYWOQ9wYFWRC6pHlVAG0H/4Clq+vHDBITAMGkQejPmRC6on1VgVKBn0LxjOoJj9BAwaRh6IWMiH1wFZhgHbZQYja+CJg0zhIwnyFjkQaACrIhDgZb7NB8/l+cIEKeE9PhqxTiNCRSANBBZkQJ7s3YBBnEMF7ZEeh45AGhAoyIU6kXXp3wKBCE1QzugkdhzQwVJAJcRLT/uswfH8GfL4ePi8/RQMGEYfRXRaEOIHlUjHuDFsFUYAS6qWjIJJLhY5EGiD6E07czs2bN9G7d2/ExsYiLi4On376qdCRamUr1EC76gi41j4I2DIO4gCl0JFIA0UtZOJ2JBIJPvroIyQmJkKj0SApKQmpqamIjXW/GTV4kwWar3+BKFgJ37m9IYmiAYPIo6MWMnE7ISEhSExMBACo1WrExMQgLy9P4FTV8TyPioV7IQpRQiRVQtG/ndCRSANHBZm4tZycHBw/fhwpKSlCR6lG++UvEAXJwe5YoZ7UWeg4pBGggkzcllarxYgRI/DJJ5/Ax6f6QO6LFi1CcnIykpOTUVRU5NJs+g3nYDmdXzlg0EwaMIg4B8cYc2R9h1Ym5FFZLBYMGjQI/fv3xyuvvPLQ9ZOTk5Gdne2CZID5RD6Kuy+GJKYZAvZMhFjt5ZL9kgaNq8tK1EImbocxhqlTpyImJqZOxdiVLNdKoF2dDXFrXwRsHEvFmDgVFWTidn7++WesWLECe/bsQUJCAhISErB161ahY8FWYYBu9VGIW6jht2gwxCFqoSORRoZueyNup3v37nCwK63e2QcMCvKCxNcP8idaCh2JNELUQiakDjSf7IMoRAGRSQzvtHih45BGigoyIQ+hXXIMvEYPvtAE7z8+IXQc0ohRQSakFqa9OSifvhm2Q3dowCBS7+jTRcgDmE/dgnb1UUjaNYX/qjQaMIjUO7qoR0gNbAUV0O88A0lME6hffBIiP5oPj9Q/KsiE/A5vMEOz5BC4ADlkrYMhax8sdCTiIaggE3IfnudR8eleiIIVEHFeUKS2FToS8SDUh0zIfbQLfwG8RMAdG1Q0YBBxMSrIhNxlWH8Omlm7wGeXQjWzh9BxiAeiLgtCABgPXIPum6OQdg5Dk/89B04sFjoS8UBUkInHs1wpgfHoVUi6BUP1URI4Bd3eRoRBBZl4NFu5Abq1R8H5SOHVKQKSFk2EjkQ8GBVk4rF4mw2af1cOGCRt0gTyrjRgEBEWXdQjHksz5ydwzeQQmcVQDu8gdBxCqCATz6T7+hi0cw6AndUf/wZJAAAQQklEQVTBezoNGETcA3VZEI9j2HoO2q+zIE+NgO+C/uBowCDiJqggE49iOpkH87V8yIdFQj21Gzgp3d5G3Ac1DYjHsOZXwPDjWYAHlAM7QOzvLXQkQqqgFjLxCLzBDO3SuwMGRYRAFkcDBhH3Qy1k0ugxxlDxzi6IghUQe3lD0beN0JEIqREVZNLoaRf8DN2/soA8K1QTk4WOQ8gDUUEmjZruuxPQfnkYitFxUL/ZS+g4hNSKCjJptIz7r8FaVgav5+Pgt/g5cBwndCRCakUX9UijZLlcBOOxqwDPoJ6YApFSJnQkQh6KCjJpdGxleujWHQenlsIrKZIGDCINBnVZkEaF2XhUvL8HXIAXpCHNIO/SQuhIhNQZFWTSqFS8/iP0Hx+FyCSFcmh7oeMQ4hAqyMQtbd++HW3btkVUVBTmz59fp220Xx2G7n9H4f1iMtQzaMAg0vBQQSZux2az4aWXXsK2bdtw9uxZZGRk4OzZs7Vuw5cbYYMRij93gu8nz7goKSHO5ZKLeqY9V8FrzNUXMPbgjWpZ9MDtnLwNe5R87nBMtW1Xyza1vZ2rzjkYw5UrVzHZqxuC9pTCglK8FTkSZ95ah5YDDL97GwaAB+NtYCYLWKkZPn/pAU5CAwaRhomrtej8zjPPPMOKi4sd3onlzG3AaHV4O0LgJQHEHDiJCJBwgEQE6KxgWjMg5iBqrgYAXLiVg5g2bcF5udd8eEVFRWjWrJnQMaqhXI553FxHjx7dwRh76Fc3hwoyam/bPJDl7G2MSxuDb775poYED96u1hv5H7TIwW1Gjx6D775b9eDtanuWoB7zjRg+At+v+/7uJo+wn9r29RjHNGTwEGzctNGhbWrbl624ArzBDGY0gxktYAYLckoLsHj3RixY8AH0u0/bP3WcTALOSwZJmD+k4QFgjMFWVAHOS4qA5ChodNpadiaM5ORkZGdnCx2jGsrlGCfkqtNTSS7pspDGBuKasgKyTiGu2J1Dbig0kLYPEjpGNbe8dJC2aSp0jGpuyw2QtKr9vl7GmP2PiOV6CfgyHXidCUxrAq8zQeSnhLJ3DADAcOgCmNZU2dr19gLnLYfMyw9n7uRAEu4H5YB4LFq2GBqbCa//7Y1q+5I09wUA8I/WViDErdCDIcRhzGqz99NarpfAVlRRWXB1JvBaEzi5BKrnEgEA5jO5sBVWgJNLwKm8IPJRQNxUZX8vZWr7ymVeUnsRj7Bacen/XcK1a9cQFhaG/36zFN9++63Lj5MQV3NZQX7hhRdctSuHUK6qGGNgBjNESjkAwHKzBNbcUnux3T7jQ2i+y4LP+Mrbyiw5RbBeKwbnLYdIJYck2BciP6X9/RS9YsBJxQ+cmUN837r3SCQSfP755+jfvz9sNhumTJmCuLi4WnM3bep+3yYA+nw5ytNzuaQPmbgPZrVVdhuovcCJRLDmlcJy5Tb4uwWX6U2V4z9MeAKcVALjkWswXyyA6G7BvVd4Ze2bg+M4MIsNEIvAiYQduMdd+x4Juct9+pCJazDGKi+M6UwQ+SjAySSwFpTDfCbvbh+uEcxUebeL97AkiP2U4DUGWAvKIfKWQxyohsi7KUQqOe59fuRJreDVufUD90lz0hHiPFSQGxBm4+1dByJfBUTecthKtDBmX7NfMIONBwAo+7WHJKwJmMUGvlwPTuUFcYCqspWr8oJIUXl7mKxdKGTtQh+4T6FbvoR4Eqc+qbdmzRrExcVBJBJV+/o4b948REVFoW3bttixY0eN21+7dg0pKSmIiorC6NGjYTbX8DDJYxo9ejQSEhKQkJCAVq1aISEhocb1WrVqhQ4dOiAhIQHJyfU/y8S7776L2Kg2SOv7LP40/A/4aeN2AACvMUC3+QQ0qw5Ds/xnaL/Phn7Hacya+MfKR4o5DrDYIPL3hiwmBF4pEVD0iYUooHICT2m4P1TDk+Hdrz0UT0ZD3rEFZJGB4OR1u1931qxZaNeuHeLj4zFs2DCUlZXVuJ6rztfDHqk2mUwYPXo0oqKikJKSgpycnHrLcs/NmzfRu3dvxMbGIi4uDp9++mm1dTIzM+Hr62v/7M2ZM6fecwEP/70wxvDnP/8ZUVFRiI+Px7Fjx+o904ULF+znISEhAT4+Pvjkk0+qrOOq8zVlyhQEBgaiffvfxj25c+cOUlNTER0djdTUVJSWlta47bJlyxAdHY3o6GgsW7bMOYEYY478q9XZs2fZ+fPn2VNPPcWOHDlif/3MmTMsPj6eGY1GdvXqVRYREcGsVmu17UeOHMkyMjIYY4xNnz6dffHFFw/b5WN55ZVX2D/+8Y8al7Vs2ZIVFRU5bV+8jWc2jYFZ8suY+XIhs+SXVb5usjDNumxW9NUeVr54n/2f4VgOY4wxm8HMtNtPMf3+C0x/9Br7y9CJ7Eb2WWYs17L4+Hh25swZp2WsyY4dO5jFYmGMMfbaa6+x1157rcb1nH2+amK1WllERAS7cuUKM5lMVY4/KSmJMcbYv//9bzZ9+nTGGGMZGRls1KhR9ZqJMcZu3brFjh49yhhjrKKigkVHR1f7vfz000/s2Wefrfcsv/ew38uWLVvYM888w3ieZwcPHmRdunRxYbrK32lQUBDLycmp8rqrztfevXvZ0aNHWVxcnP21WbNmsXnz5jHGGJs3b16Nn/mSkhLWunVrVlJSwu7cucNat27N7ty5U9uu6lRjndpCjomJQdu2bau9vmHDBowZMwZyuRytW7dGVFQUsrKyqv1h2LNnD9LS0gAA6enpWL9+vTPjVdvf6tWrMXbsWOe8n9kKW6kOlpt3YD5/C5acYvt+tGuPQLP8ALRrjkC/7RQM+y7AcrmwckOpGCI/JX7V3sJB3XUoesfAe3AC5LGV3QgiLym8+3eAonsbnDDl45y+AOFJMZD7eGPMmDHYsGGDU/I/SL9+/SCRVPZsde3aFbm5ufW6v9pkZWUhKioKERERkMlkNR7/hg0bkJ6eDgBIS0vD7t27a38E3glCQkKQmFh5m59arUZMTAzy8vLqdZ/OsmHDBkycOBEcx6Fr164oKytDfn6+y/a/e/duREZGomXLli7b5/169uwJf3//Kq/d/xl6UB3asWMHUlNT4e/vjyZNmiA1NRXbt29/7DwuGVwoLy8P4eHh9p+bN29e7QNbUlICPz8/+3/8Na3jTPv370dQUBCio6NrXM5xHPr164ekpCQsWrQIvN4E6+0KWK4WwXT6Jsxnb9nX1W44Bs03B6FbfwyGXWdgPHgF5osF9veRtAiALD4cXk9GQ9mvPbyHJ8Gra6R9ubJ3DDJLL2H6h28haUgfTHvtLyjTV3/qrC7nsT4tXrwYAwYMqHHZ789XfajL8d+/jkQiga+vL0pKSuolT01ycnJw/PhxpKSkVFt28OBBdOzYEQMGDMCZM2dckudhvxehP1OrVq16YKNIiPMFAIWFhQgJqXyILTg4GIWFhdXWqa/z5vBFvb59+6KgoKDa6++//z6ee+65xw7kDHXJmJGRYf8gMKut8pYvnenu7V9GZC3egGa943H79m2c+XIjtN9VbdGLmqogu9uKlUYFATyDyFsOTnX39jCv36YM8uoS8dBcL774ImbPng2O4zB79my8+uqrWLx4sXNOyEPU5Xy9//77kEgkGD9+fI3vceDAAYSFheH27dtITU1Fu3bt0LNnz3rN7W60Wi1GjBiBTz75BD4+PlWWJSYm4vr161CpVNi6dSuGDh2KS5cu1Xsmd/69mM1mbNy4EfPmzau2TKjz9Xscx7l0LkaHC/KuXbsc3klYWBhu3rxp/zk3NxdhYWFV1gkICEBZWRmsViskEkmN6zxqRmaygK8wgteZYDqTB1uFHj1kLdBz1CgAgPGXy7Bcuf3bBhyg8FEAAAIDA7FLYcZ+7TX0GzoIIpW8svDKfjt18ri65azruZs2bRoGDRpU7fW6nMdH8bBcS5cuxebNm7F79+4Hfjjv5QgMDMSwYcOQlZXl9P/w63L899Zp3rw5rFYrysvLERAQ4NQcNbFYLBgxYgTGjx+P4cOHV1t+f4EeOHAgZsyYgeLi4np/oOVhv5f6+kzVxbZt25CYmIigoOpDFwh1vgAgKCgI+fn5CAkJQX5+PgIDA6utExYWhszMTPvPubm56NWr12Pv2yVdFkOGDMGqVatgMplw7do1XLp0CV26dKmyDsdx6N27N9auXQug8gpmXVvcvNEM660ymC8VwnT8OgwHLkK3/TSYufKeW9OvedBtPgHDT+dgyroK04V8xLeMRljI3RZudBC8eraFckA8VCM7QzSiE1hqZV+4TqfD5+tXQhwdCGm4P8RNvKsUY2e5v9/uhx9+qHLV957OnTvj0qXKR4rNZjNWrVqFIUOGOD3L/bZv344FCxZg48aNUCqrP1UHVJ4jjUZj//87d+6sMf/jqsvxDxkyxH7Fe+3atejTp0+9t3AYY5g6dSpiYmLwyiuv1LhOQUGBvS87KysLPM/X+x+KuvxehgwZguXLl4MxhkOHDsHX19f+db2+3f8t9feEOF/33P8ZelAd6t+/P3bu3InS0lKUlpZi586d6N+//+PvvK5X/1gd7rJYt24dCwsLYzKZjAUGBrJ+/frZl82dO5dFRESwNm3asK1bt9pfHzBgAMvLy2OMMXblyhXWuXNnFhkZydLS0pjRaGSMMWbTm5g5p5gZz+Qyw+ErTLfnLNNsPM6spTrGGGOmc3lV7lCoyDjEtJuOM5vGwBhjzHpHy8zXi5m1WMN4o5mlp6ezL7/8skr2vLw8NmDAAHuO+Ph4Fh8fz2JjY9ncuXMfduiPbcKECax9+/asQ4cObPDgwezWrVvVcjFWeVU8OjqaRUREuCRXZGQka968OevYsSPr2LGj/Q4Goc5XTcc/e/ZsFhkZyRhjzGAwsLS0NBYZGck6d+7Mrly5Um9Z7tm/fz8DwDp06GA/T1u2bGFffvml/XP22WefsdjYWBYfH89SUlLYzz//XO+5HvR7uT8Xz/NsxowZLCIigrVv377K3VH1SavVMn9/f1ZWVmZ/TYjzNWbMGBYcHMwkEgkLCwtjX331FSsuLmZ9+vRhUVFR7Omnn2YlJSWMMcaOHDnCpk6dat/266+/ZpGRkSwyMpItXrz4YbuqU411i0eneaMF1rxSMK0R/N0HHJjOBK+UCEhCm8ByswSGXXdnjBCL7H21Xp0jIPb3rtxGY6h83VsOTkwToXgaenSauDn3eXSama2wXL39W7G9+7/yhBaQtQkG05lg3HcBAKqMCgZx5WO5kkBfeA9OqCy2940Kdo/o7oU0QghpyFxTkHkG48ErgIirHJzG++6oYHeLqMhPCe9hSZV3J9Qw/Q4nl0AsV7siKiGECMYlBZmTS6Aa3QWcQlbjBRZOLKpxGEZCCPEkrinIHAdOSV0KhBBSG7r6RQghboIKMiGEuAkqyIQQ4iaoIBNCiJuggkwIIW6CCjIhhLgJKsiEEOImHB3LghC3xHHcdsbYM0LnIORxUEEmhBA3QV0WhBDiJqggE0KIm6CCTAghboIKMiGEuAkqyIQQ4iaoIBNCiJuggkwIIW6CCjIhhLgJKsiEEOIm/j8shOcNZC76hQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ReLU\n",
    "def relu(z):\n",
    "    return paddle.maximum(z, paddle.to_tensor(0.))\n",
    "\n",
    "# 带泄露的ReLU\n",
    "def leaky_relu(z, negative_slope=0.1):\n",
    "    # 当前版本paddle暂不支持直接将bool类型转成int类型，因此调用了paddle的cast函数来进行显式转换\n",
    "    a1 = (paddle.cast((z > 0), dtype='float32') * z) \n",
    "    a2 = (paddle.cast((z <= 0), dtype='float32') * (negative_slope * z))\n",
    "    return a1 + a2\n",
    "\n",
    "# 在[-10,10]的范围内生成一系列的输入值，用于绘制relu、leaky_relu的函数曲线\n",
    "z = paddle.linspace(-10, 10, 10000)\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(z.tolist(), relu(z).tolist(), color=\"#e4007f\", label=\"ReLU Function\")\n",
    "plt.plot(z.tolist(), leaky_relu(z).tolist(), color=\"#f19ec2\", linestyle=\"--\", label=\"LeakyReLU Function\")\n",
    "\n",
    "ax = plt.gca()\n",
    "ax.spines['top'].set_color('none')\n",
    "ax.spines['right'].set_color('none')\n",
    "ax.spines['left'].set_position(('data',0))\n",
    "ax.spines['bottom'].set_position(('data',0))\n",
    "plt.legend(loc='upper left', fontsize='large')\n",
    "plt.savefig('fw-relu-leakyrelu.pdf')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**********\n",
    "\n",
    "**说明**\n",
    "\n",
    "在飞桨中，可以通过调用`paddle.nn.functional.relu`和`paddle.nn.functional.leaky_relu`完成ReLU与带泄露的ReLU的计算。\n",
    "\n",
    "**********\n",
    "\n",
    "\n",
    "**动手练习**\n",
    "本节重点介绍和实现了几个经典的Sigmoid函数和ReLU函数。\n",
    "请动手实现《神经网络与深度学习》4.1节中提到的其他激活函数，如：Hard-Logistic、Hard-Tanh、ELU、Softplus、Swish等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.2 基于前馈神经网络的二分类任务\n",
    "\n",
    "前馈神经网络的网络结构如**图4.3**所示。每一层获取前一层神经元的活性值，并重复上述计算得到该层的活性值，传入到下一层。整个网络中无反馈，信号从输入层向输出层逐层的单向传播，得到网络最后的输出 $\\boldsymbol{a}^{(L)}$。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/dbcf147a4e00446792eb2b93834e0f3154936e08ea124242af8631fde204381c\" width=\"500\" hegiht=\"\" ></center>\n",
    "<center><br>图4.3: 前馈神经网络结构</br></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.2.1 数据集构建\n",
    "\n",
    "这里，我们使用第3.1.1节中构建的二分类数据集：Moon1000数据集，其中训练集640条、验证集160条、测试集200条。\n",
    "该数据集的数据是从两个带噪音的弯月形状数据分布中采样得到，每个样本包含2个特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "outer_circ_x.shape: [500] outer_circ_y.shape: [500]\n",
      "inner_circ_x.shape: [500] inner_circ_y.shape: [500]\n",
      "after concat shape: [1000]\n",
      "X shape: [1000, 2]\n",
      "y shape: [1000]\n"
     ]
    }
   ],
   "source": [
    "from nndl.dataset import make_moons\n",
    "\n",
    "# 采样1000个样本\n",
    "n_samples = 1000\n",
    "X, y = make_moons(n_samples=n_samples, shuffle=True, noise=0.5)\n",
    "\n",
    "num_train = 640\n",
    "num_dev = 160\n",
    "num_test = 200\n",
    "\n",
    "X_train, y_train = X[:num_train], y[:num_train]\n",
    "X_dev, y_dev = X[num_train:num_train + num_dev], y[num_train:num_train + num_dev]\n",
    "X_test, y_test = X[num_train + num_dev:], y[num_train + num_dev:]\n",
    "\n",
    "y_train = y_train.reshape([-1,1])\n",
    "y_dev = y_dev.reshape([-1,1])\n",
    "y_test = y_test.reshape([-1,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.2.2 模型构建\n",
    "\n",
    "为了更高效的构建前馈神经网络，我们先定义每一层的算子，然后再通过算子组合构建整个前馈神经网络。\n",
    "\n",
    "假设网络的第$l$层的输入为第$l-1$层的神经元活性值$\\boldsymbol{a}^{(l-1)}$，经过一个仿射变换，得到该层神经元的净活性值$\\boldsymbol{z}$，再输入到激活函数得到该层神经元的活性值$\\boldsymbol{a}$。\n",
    "\n",
    "在实践中，为了提高模型的处理效率，通常将$N$个样本归为一组进行成批地计算。假设网络第$l$层的输入为$\\boldsymbol{A}^{(l-1)}\\in \\mathbb{R}^{N\\times M_{l-1}}$，其中每一行为一个样本，则前馈网络中第$l$层的计算公式为\n",
    "\n",
    "$$\n",
    "\\mathbf Z^{(l)}=\\mathbf A^{(l-1)} \\mathbf W^{(l)} +\\mathbf b^{(l)}  \\in \\mathbb{R}^{N\\times M_{l}}, (4.8)\n",
    "$$\n",
    "$$\n",
    "\\mathbf A^{(l)}=f_l(\\mathbf Z^{(l)}) \\in \\mathbb{R}^{N\\times M_{l}}, (4.9)\n",
    "$$\n",
    "其中$\\mathbf Z^{(l)}$为$N$个样本第$l$层神经元的净活性值，$\\mathbf A^{(l)}$为$N$个样本第$l$层神经元的活性值，$\\boldsymbol{W}^{(l)}\\in \\mathbb{R}^{M_{l-1}\\times M_{l}}$为第$l$层的权重矩阵，$\\boldsymbol{b}^{(l)}\\in \\mathbb{R}^{1\\times M_{l}}$为第$l$层的偏置。\n",
    "\n",
    "***\n",
    "为了和代码的实现保存一致性，这里使用形状为$(样本数量\\times 特征维度)$的张量来表示一组样本。样本的矩阵$\\boldsymbol{X}$是由$N$个$\\boldsymbol{x}$的**行向量**组成。而《神经网络与深度学习》中$\\boldsymbol{x}$为列向量，因此这里的权重矩阵$\\boldsymbol{W}$和偏置$\\boldsymbol{b}$和《神经网络与深度学习》中的表示刚好为转置关系。\n",
    "***\n",
    "\n",
    "为了使后续的模型搭建更加便捷，我们将神经层的计算，即公式(4.8)和(4.9)，都封装成算子，这些算子都继承`Op`基类。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.2.2.1 线性层算子\n",
    "公式（4.8）对应一个线性层算子，权重参数采用默认的随机初始化，偏置采用默认的零初始化。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nndl.op import Op\r\n",
    "\r\n",
    "# 实现线性层算子\r\n",
    "class Linear(Op):\r\n",
    "    def __init__(self, input_size, output_size, name, weight_init=paddle.standard_normal, bias_init=paddle.zeros):\r\n",
    "        \"\"\"\r\n",
    "        输入：\r\n",
    "            - input_size：输入数据维度\r\n",
    "            - output_size：输出数据维度\r\n",
    "            - name：算子名称\r\n",
    "            - weight_init：权重初始化方式，默认使用'paddle.standard_normal'进行标准正态分布初始化\r\n",
    "            - bias_init：偏置初始化方式，默认使用全0初始化\r\n",
    "        \"\"\"\r\n",
    "        \r\n",
    "        self.params = {}\r\n",
    "        # 初始化权重\r\n",
    "        self.params['W'] = weight_init(shape=[input_size,output_size])\r\n",
    "        # 初始化偏置\r\n",
    "        self.params['b'] = bias_init(shape=[1,output_size])\r\n",
    "        self.inputs = None\r\n",
    "\r\n",
    "        self.name = name\r\n",
    "\r\n",
    "    def forward(self, inputs):\r\n",
    "        \"\"\"\r\n",
    "        输入：\r\n",
    "            - inputs：shape=[N,input_size], N是样本数量\r\n",
    "        输出：\r\n",
    "            - outputs：预测值，shape=[N,output_size]\r\n",
    "        \"\"\"\r\n",
    "        self.inputs = inputs\r\n",
    "\r\n",
    "        outputs = paddle.matmul(self.inputs, self.params['W']) + self.params['b']\r\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.2.2.2 Logistic算子\n",
    "\n",
    "本节我们采用Logistic函数来作为公式(4.9)中的激活函数。这里也将Logistic函数实现一个算子，代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Logistic(Op):\r\n",
    "    def __init__(self):\r\n",
    "        self.inputs = None\r\n",
    "        self.outputs = None\r\n",
    "\r\n",
    "    def forward(self, inputs):\r\n",
    "        \"\"\"\r\n",
    "        输入：\r\n",
    "            - inputs: shape=[N,D]\r\n",
    "        输出：\r\n",
    "            - outputs：shape=[N,D]\r\n",
    "        \"\"\"\r\n",
    "        outputs = 1.0 / (1.0 + paddle.exp(-inputs))\r\n",
    "        self.outputs = outputs\r\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.2.2.3 层的串行组合\n",
    "\n",
    "在定义了神经层的线性层算子和激活函数算子之后，我们可以不断交叉重复使用它们来构建一个多层的神经网络。\n",
    "\n",
    "下面我们实现一个两层的用于二分类任务的前馈神经网络，选用Logistic作为激活函数，可以利用上面实现的线性层和激活函数算子来组装。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 实现一个两层前馈神经网络\r\n",
    "class Model_MLP_L2(Op):\r\n",
    "    def __init__(self, input_size, hidden_size, output_size):\r\n",
    "        \"\"\"\r\n",
    "        输入：\r\n",
    "            - input_size：输入维度\r\n",
    "            - hidden_size：隐藏层神经元数量\r\n",
    "            - output_size：输出维度\r\n",
    "        \"\"\"\r\n",
    "        self.fc1 = Linear(input_size, hidden_size, name=\"fc1\")\r\n",
    "        self.act_fn1 = Logistic()\r\n",
    "        self.fc2 = Linear(hidden_size, output_size, name=\"fc2\")\r\n",
    "        self.act_fn2 = Logistic()\r\n",
    "\r\n",
    "    def __call__(self, X):\r\n",
    "        return self.forward(X)\r\n",
    "\r\n",
    "    def forward(self, X):\r\n",
    "        \"\"\"\r\n",
    "        输入：\r\n",
    "            - X：shape=[N,input_size], N是样本数量\r\n",
    "        输出：\r\n",
    "            - a2：预测值，shape=[N,output_size]\r\n",
    "        \"\"\"\r\n",
    "        z1 = self.fc1(X)\r\n",
    "        a1 = self.act_fn1(z1)\r\n",
    "        z2 = self.fc2(a1)\r\n",
    "        a2 = self.act_fn2(z2)\r\n",
    "        return a2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**测试一下**\n",
    "\n",
    "现在，我们实例化一个两层的前馈网络，令其输入层维度为5，隐藏层维度为10，输出层维度为1。\n",
    "并随机生成一条长度为5的数据输入两层神经网络，观察输出结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result:  Tensor(shape=[1, 1], dtype=float32, place=CPUPlace, stop_gradient=True,\n",
      "       [[0.46585333]])\n"
     ]
    }
   ],
   "source": [
    "# 实例化模型\r\n",
    "model = Model_MLP_L2(input_size=5, hidden_size=10, output_size=1)\r\n",
    "# 随机生成1条长度为5的数据\r\n",
    "X = paddle.rand(shape=[1, 5])\r\n",
    "result = model(X)\r\n",
    "print (\"result: \", result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.2.3 损失函数\n",
    "二分类交叉熵损失函数见第三章，这里不再赘述。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.2.4 模型优化\n",
    "神经网络的参数主要是通过**梯度下降法**进行优化的，因此需要计算最终损失对每个参数的梯度。\n",
    "由于神经网络的层数通常比较深，其梯度计算和上一章中的线性分类模型的不同的点在于：线性模型通常比较简单可以直接计算梯度，而神经网络相当于一个复合函数，需要利用链式法则进行反向传播来计算梯度。\n",
    "\n",
    "#### 4.2.4.1 反向传播算法\n",
    "\n",
    "前馈神经网络的参数梯度通常使用**误差反向传播**算法来计算。使用误差反向传播算法的前馈神经网络训练过程可以分为以下三步：\n",
    "\n",
    "1. 前馈计算每一层的净活性值$\\boldsymbol{Z}^{(l)}$和激活值$\\boldsymbol{A}^ {(l)}$，直到最后一层；\n",
    "1. 反向传播计算每一层的误差项$\\delta^{(l)}=\\frac{\\partial R}{\\partial \\boldsymbol{Z}^{(l)}}$；\n",
    "1. 计算每一层参数的梯度，并更新参数。\n",
    "\n",
    "在上面实现算子的基础上，来实现误差反向传播算法。在上面的三个步骤中，\n",
    "1. 第1步是前向计算，可以利用算子的`forward()`方法来实现；\n",
    "1. 第2步是反向计算梯度，可以利用算子的`backward()`方法来实现；\n",
    "1. 第3步中的计算参数梯度也放到`backward()`中实现，更新参数放到另外的**优化器**中专门进行。\n",
    "\n",
    "这样，在模型训练过程中，我们首先执行模型的`forward()`，再执行模型的`backward()`，就得到了所有参数的梯度，之后再利用优化器迭代更新参数。\n",
    "\n",
    "以这我们这节中构建的两层全连接前馈神经网络`Model_MLP_L2`为例，下图给出了其前向和反向计算过程：\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/8562dfb10d464396948d05ee3620cec1d057025dddee43ff92dae3fbb72e8f65)\n",
    "\n",
    "下面我们按照反向的梯度传播顺序，为每个算子添加`backward()`方法，并在其中实现每一层参数的梯度的计算。\n",
    "\n",
    "#### 4.2.4.2 损失函数\n",
    "\n",
    "二分类交叉熵损失函数对神经网络的输出$\\hat{\\boldsymbol{y}}$的偏导数为:\n",
    "$$\n",
    "\\frac{\\partial R}{\\partial \\hat{\\boldsymbol{y}}} =  -\\frac{1}{N}(\\mathrm{dialog}(\\frac{1}{\\hat{\\boldsymbol{y}}})\\boldsymbol{y}-\\mathrm{dialog}(\\frac{1}{1-\\hat{\\boldsymbol{y}}})(1-\\boldsymbol{y})) (4.10) \\\\ \n",
    "= -\\frac{1}{N}(\\frac{1}{\\hat{\\boldsymbol{y}}}\\odot\\boldsymbol{y}-\\frac{1}{1-\\hat{\\boldsymbol{y}}}\\odot(1-\\boldsymbol{y})), (4.11)\n",
    "$$\n",
    "其中$dialog(\\boldsymbol{x})$表示以向量$\\boldsymbol{x}$为对角元素的对角阵，$\\frac{1}{\\boldsymbol{x}}=\\frac{1}{x_1},...,\\frac{1}{x_N}$表示逐元素除，$\\odot$表示逐元素积。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "实现损失函数的`backward()`，代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 实现交叉熵损失函数\n",
    "class BinaryCrossEntropyLoss(Op):\n",
    "    def __init__(self, model):\n",
    "        self.predicts = None\n",
    "        self.labels = None\n",
    "        self.num = None\n",
    "\n",
    "        self.model = model\n",
    "\n",
    "    def __call__(self, predicts, labels):\n",
    "        return self.forward(predicts, labels)\n",
    "\n",
    "    def forward(self, predicts, labels):\n",
    "        \"\"\"\n",
    "        输入：\n",
    "            - predicts：预测值，shape=[N, 1]，N为样本数量\n",
    "            - labels：真实标签，shape=[N, 1]\n",
    "        输出：\n",
    "            - 损失值：shape=[1]\n",
    "        \"\"\"\n",
    "        self.predicts = predicts\n",
    "        self.labels = labels\n",
    "        self.num = self.predicts.shape[0]\n",
    "        loss = -1. / self.num * (paddle.matmul(self.labels.t(), paddle.log(self.predicts)) \n",
    "                + paddle.matmul((1-self.labels.t()), paddle.log(1-self.predicts)))\n",
    "\n",
    "        loss = paddle.squeeze(loss, axis=1)\n",
    "        return loss\n",
    "\n",
    "    def backward(self):\n",
    "        # 计算损失函数对模型预测的导数\n",
    "        loss_grad_predicts = -1.0 * (self.labels / self.predicts - \n",
    "                       (1 - self.labels) / (1 - self.predicts)) / self.num\n",
    "        \n",
    "        # 梯度反向传播\n",
    "        self.model.backward(loss_grad_predicts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.2.4.3 Logistic算子\n",
    "在本节中，我们使用Logistic激活函数，所以这里为Logistic算子增加的反向函数。\n",
    "\n",
    "Logistic算子的前向过程表示为$\\boldsymbol{A}=\\sigma(\\boldsymbol{Z})$，其中$\\sigma$为Logistic函数，$\\boldsymbol{Z} \\in R^{N \\times D}$和$\\boldsymbol{A} \\in R^{N \\times D}$的每一行表示一个样本。\n",
    "\n",
    "为了简便起见，我们分别用向量$\\boldsymbol{a} \\in R^D$ 和 $\\boldsymbol{z} \\in R^D$表示同一个样本在激活函数前后的表示，则$\\boldsymbol{a}$对$\\boldsymbol{z}$的偏导数为：\n",
    "$$\n",
    "\\frac{\\partial \\boldsymbol{a}}{\\partial \\boldsymbol{z}}=diag(\\boldsymbol{a}\\odot(1-\\boldsymbol{a}))\\in R^{D \\times D}, (4.12)\n",
    "$$\n",
    "按照反向传播算法，令$\\delta_{\\boldsymbol{a}}=\\frac{\\partial R}{\\partial \\boldsymbol{a}} \\in R^D$表示最终损失$R$对Logistic算子的单个输出$\\boldsymbol{a}$的梯度，则\n",
    "$$\n",
    "\\delta_{\\boldsymbol{z}} \\triangleq \\frac{\\partial R}{\\partial \\boldsymbol{z}} = \\frac{\\partial \\boldsymbol{a}}{\\partial \\boldsymbol{z}}\\delta_{\\boldsymbol{a}}  (4.13) \\\\\n",
    "= diag(\\boldsymbol{a}\\odot(1-\\boldsymbol{a}))\\delta_{\\boldsymbol(a)}, (4.14) \\\\\n",
    "= \\boldsymbol{a}\\odot(1-\\boldsymbol{a})\\odot\\delta_{\\boldsymbol(a)}。 (4.15)\n",
    "$$\n",
    "\n",
    "将上面公式利用批量数据表示的方式重写，令$\\delta_{\\boldsymbol{A}} =\\frac{\\partial R}{\\partial \\boldsymbol{A}} \\in R^{N \\times D}$表示最终损失$R$对Logistic算子输出$A$的梯度，损失函数对Logistic函数输入$\\boldsymbol{Z}$的导数为\n",
    "$$\n",
    "\\delta_{\\boldsymbol{Z}}=\\boldsymbol{A} \\odot (1-\\boldsymbol{A})\\odot \\delta_{\\boldsymbol{A}} \\in R^{N \\times D},(4.16)\n",
    "$$\n",
    "$\\delta_{\\boldsymbol{Z}}$为Logistic算子反向传播的输出。\n",
    "\n",
    "由于Logistic函数中没有参数，这里不需要在`backward()`方法中计算该算子参数的梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Logistic(Op):\n",
    "    def __init__(self):\n",
    "        self.inputs = None\n",
    "        self.outputs = None\n",
    "        self.params = None\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        outputs = 1.0 / (1.0 + paddle.exp(-inputs))\n",
    "        self.outputs = outputs\n",
    "        return outputs\n",
    "\n",
    "    def backward(self, grads):\n",
    "        # 计算Logistic激活函数对输入的导数\n",
    "        outputs_grad_inputs = paddle.multiply(self.outputs, (1.0 - self.outputs))\n",
    "        return paddle.multiply(grads,outputs_grad_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.2.4.4 线性层\n",
    "\n",
    "线性层算子Linear的前向过程表示为$\\boldsymbol{Y}=\\boldsymbol{X}\\boldsymbol{W}+\\boldsymbol{b}$，其中输入为$\\boldsymbol{X} \\in R^{N \\times M}$，输出为$\\boldsymbol{Y} \\in R^{N \\times D}$，参数为权重矩阵$\\boldsymbol{W} \\in R^{M \\times D}$和偏置$\\boldsymbol{b} \\in R^{1 \\times D}$。$\\boldsymbol{X}$和$\\boldsymbol{Y}$中的每一行表示一个样本。\n",
    "\n",
    "为了简便起见，我们用向量$\\boldsymbol{x}\\in R^M$和$\\boldsymbol{y}\\in R^D$表示同一个样本在线性层算子中的输入和输出，则有$\\boldsymbol{y}=\\boldsymbol{W}^T\\boldsymbol{x}+\\boldsymbol{b}^T$。$\\boldsymbol{y}$对输入$\\boldsymbol{x}$的偏导数为\n",
    "$$\n",
    "\\frac{\\partial \\boldsymbol{y}}{\\partial \\boldsymbol{x}} = \\boldsymbol{W}\\in R^{D \\times M}。(4.17)\n",
    "$$\n",
    "\n",
    "**线性层输入的梯度** 按照反向传播算法，令$\\delta_{\\boldsymbol{y}}=\\frac{\\partial R}{\\partial \\boldsymbol{y}}\\in R^D$表示最终损失$R$对线性层算子的单个输出$\\boldsymbol{y}$的梯度，则\n",
    "$$\n",
    "\\delta_{\\boldsymbol{x}} \\triangleq \\frac{\\partial R}{\\partial \\boldsymbol{x}}= \\boldsymbol{W} \\delta_{\\boldsymbol{y}}。(4.18)\n",
    "$$\n",
    "\n",
    "将上面公式利用批量数据表示的方式重写，令$\\delta_{\\boldsymbol{Y}}=\\frac{\\partial R}{\\partial \\boldsymbol{Y}}\\in \\mathbb{R}^{N\\times D}$表示最终损失$R$对线性层算子输出$\\boldsymbol{Y}$的梯度，公式可以重写为\n",
    "$$\n",
    "\\delta_{\\boldsymbol{X}} =\\delta_{\\boldsymbol{Y}} \\boldsymbol{W}^T,(4.19)\n",
    "$$\n",
    "其中$\\delta_{\\boldsymbol{X}}$为线性层算子反向函数的输出。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**计算线性层参数的梯度** 由于线性层算子中包含有可学习的参数$\\boldsymbol{W}$和$\\boldsymbol{b}$，因此`backward()`除了实现梯度反传外，还需要计算算子内部的参数的梯度。\n",
    "\n",
    "令$\\delta_{\\boldsymbol{y}}=\\frac{\\partial R}{\\partial \\boldsymbol{y}}\\in \\mathbb{R}^D$表示最终损失$R$对线性层算子的单个输出$\\boldsymbol{y}$的梯度，则\n",
    "$$\n",
    "\\delta_{\\boldsymbol{W}} \\triangleq \\frac{\\partial R}{\\partial \\boldsymbol{W}} = \\boldsymbol{x}\\delta_{\\boldsymbol{y}}^T,(4.20) \\\\\n",
    "\\delta_{\\boldsymbol{b}} \\triangleq \\frac{\\partial R}{\\partial \\boldsymbol{b}} = \\delta_{\\boldsymbol{y}}^T。(4.21)\n",
    "$$\n",
    "\n",
    "将上面公式利用批量数据表示的方式重写，令$\\delta_{\\boldsymbol{Y}}=\\frac{\\partial R}{\\partial \\boldsymbol{Y}}\\in \\mathbb{R}^{N\\times D}$表示最终损失$R$对线性层算子输出$\\boldsymbol{Y}$的梯度，则公式可以重写为\n",
    "$$\n",
    "\\delta_{\\boldsymbol{W}} = \\boldsymbol{X}^T \\delta_{\\boldsymbol{Y}},(4.22) \\\\\n",
    "\\delta_{\\boldsymbol{b}} = \\mathbf{1}^T \\delta_{\\boldsymbol{Y}}。(4.23)\n",
    "$$\n",
    "\n",
    "具体实现代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Linear(Op):\n",
    "    def __init__(self, input_size, output_size, name, weight_init=paddle.standard_normal, bias_init=paddle.zeros):\n",
    "        self.params = {}\n",
    "        self.params['W'] = weight_init(shape=[input_size, output_size])\n",
    "        self.params['b'] = bias_init(shape=[1, output_size])\n",
    "\n",
    "        self.inputs = None\n",
    "        self.grads = {}\n",
    "\n",
    "        self.name = name\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        self.inputs = inputs\n",
    "        outputs = paddle.matmul(self.inputs, self.params['W']) + self.params['b']\n",
    "        return outputs\n",
    "\n",
    "    def backward(self, grads):\n",
    "        \"\"\"\n",
    "        输入：\n",
    "            - grads：损失函数对当前层输出的导数\n",
    "        输出：\n",
    "            - 损失函数对当前层输入的导数\n",
    "        \"\"\"\n",
    "        self.grads['W'] = paddle.matmul(self.inputs.T, grads)\n",
    "        self.grads['b'] = paddle.sum(grads, axis=0)\n",
    "\n",
    "        # 线性层输入的梯度\n",
    "        return paddle.matmul(grads, self.params['W'].T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.2.4.5 整个网络\n",
    "\n",
    "实现完整的两层神经网络的前向和反向计算。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Model_MLP_L2(Op):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        # 线性层\n",
    "        self.fc1 = Linear(input_size, hidden_size, name=\"fc1\")\n",
    "        # Logistic激活函数层\n",
    "        self.act_fn1 = Logistic()\n",
    "        self.fc2 = Linear(hidden_size, output_size, name=\"fc2\")\n",
    "        self.act_fn2 = Logistic()\n",
    "\n",
    "        self.layers = [self.fc1, self.act_fn1, self.fc2, self.act_fn2]\n",
    "\n",
    "    def __call__(self, X):\n",
    "        return self.forward(X)\n",
    "\n",
    "    # 前向计算\n",
    "    def forward(self, X):\n",
    "        z1 = self.fc1(X)\n",
    "        a1 = self.act_fn1(z1)\n",
    "        z2 = self.fc2(a1)\n",
    "        a2 = self.act_fn2(z2)\n",
    "        return a2\n",
    "        \n",
    "    # 反向计算\n",
    "    def backward(self, loss_grad_a2):\n",
    "        loss_grad_z2 = self.act_fn2.backward(loss_grad_a2)\n",
    "        loss_grad_a1 = self.fc2.backward(loss_grad_z2)\n",
    "        loss_grad_z1 = self.act_fn1.backward(loss_grad_a1)\n",
    "        loss_grad_inputs = self.fc1.backward(loss_grad_z1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.2.4.6 优化器\n",
    "\n",
    "在计算好神经网络参数的梯度之后，我们将梯度下降法中参数的更新过程实现在优化器中。\n",
    "\n",
    "与第3章中实现的梯度下降优化器`SimpleBatchGD`不同的是，此处的优化器需要遍历每层，对每层的参数分别做更新。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nndl.opitimizer import Optimizer\r\n",
    "\r\n",
    "class BatchGD(Optimizer):\r\n",
    "    def __init__(self, init_lr, model):\r\n",
    "        super(BatchGD, self).__init__(init_lr=init_lr, model=model)\r\n",
    "\r\n",
    "    def step(self):\r\n",
    "        # 参数更新\r\n",
    "        for layer in self.model.layers: # 遍历所有层\r\n",
    "            if isinstance(layer.params, dict):\r\n",
    "                for key in layer.params.keys():\r\n",
    "                    layer.params[key] = layer.params[key] - self.init_lr * layer.grads[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.2.5 完善Runner类：RunnerV2_1\n",
    "基于3.1.6实现的 RunnerV2 类主要针对比较简单的模型。而在本章中，模型由多个算子组合而成，通常比较复杂，因此本节继续完善并实现一个改进版： `RunnerV2_1`类，其主要加入的功能有：\n",
    "1. 支持自定义算子的梯度计算，在训练过程中调用`self.loss_fn.backward()`从损失函数开始反向计算梯度；\n",
    "1. 每层的模型保存和加载，将每一层的参数分别进行保存和加载。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "class RunnerV2_1(object):\n",
    "    def __init__(self, model, optimizer, metric, loss_fn, **kwargs):\n",
    "        self.model = model\n",
    "        self.optimizer = optimizer\n",
    "        self.loss_fn = loss_fn\n",
    "        self.metric = metric\n",
    "\n",
    "        # 记录训练过程中的评估指标变化情况\n",
    "        self.train_scores = []\n",
    "        self.dev_scores = []\n",
    "\n",
    "        # 记录训练过程中的评价指标变化情况\n",
    "        self.train_loss = []\n",
    "        self.dev_loss = []\n",
    "\n",
    "    def train(self, train_set, dev_set, **kwargs):\n",
    "        # 传入训练轮数，如果没有传入值则默认为0\n",
    "        num_epochs = kwargs.get(\"num_epochs\", 0)\n",
    "        # 传入log打印频率，如果没有传入值则默认为100\n",
    "        log_epochs = kwargs.get(\"log_epochs\", 100)\n",
    "\n",
    "        # 传入模型保存路径\n",
    "        save_dir = kwargs.get(\"save_dir\", None)\n",
    "        \n",
    "        # 记录全局最优指标\n",
    "        best_score = 0\n",
    "        # 进行num_epochs轮训练\n",
    "        for epoch in range(num_epochs):\n",
    "            X, y = train_set\n",
    "            # 获取模型预测\n",
    "            logits = self.model(X)\n",
    "            # 计算交叉熵损失\n",
    "            trn_loss = self.loss_fn(logits, y) # return a tensor\n",
    "            \n",
    "            self.train_loss.append(trn_loss.item())\n",
    "            # 计算评估指标\n",
    "            trn_score = self.metric(logits, y).item()\n",
    "            self.train_scores.append(trn_score)\n",
    "\n",
    "            self.loss_fn.backward()\n",
    "\n",
    "            # 参数更新\n",
    "            self.optimizer.step()\n",
    "           \n",
    "            dev_score, dev_loss = self.evaluate(dev_set)\n",
    "            # 如果当前指标为最优指标，保存该模型\n",
    "            if dev_score > best_score:\n",
    "                print(f\"[Evaluate] best accuracy performence has been updated: {best_score:.5f} --> {dev_score:.5f}\")\n",
    "                best_score = dev_score\n",
    "                if save_dir:\n",
    "                    self.save_model(save_dir)\n",
    "\n",
    "            if log_epochs and epoch % log_epochs == 0:\n",
    "                print(f\"[Train] epoch: {epoch}/{num_epochs}, loss: {trn_loss.item()}\")\n",
    "                \n",
    "    def evaluate(self, data_set):\n",
    "        X, y = data_set\n",
    "        # 计算模型输出\n",
    "        logits = self.model(X)\n",
    "        # 计算损失函数\n",
    "        loss = self.loss_fn(logits, y).item()\n",
    "        self.dev_loss.append(loss)\n",
    "        # 计算评估指标\n",
    "        score = self.metric(logits, y).item()\n",
    "        self.dev_scores.append(score)\n",
    "        return score, loss\n",
    "    \n",
    "    def predict(self, X):\n",
    "        return self.model(X)\n",
    "\n",
    "    def save_model(self, save_dir):\n",
    "        # 对模型每层参数分别进行保存，保存文件名称与该层名称相同\n",
    "        for layer in self.model.layers: # 遍历所有层\n",
    "            if isinstance(layer.params, dict):\n",
    "                paddle.save(layer.params, os.path.join(save_dir, layer.name+\".pdparams\"))\n",
    "\n",
    "    def load_model(self, model_dir):\n",
    "        # 获取所有层参数名称和保存路径之间的对应关系\n",
    "        model_file_names = os.listdir(model_dir)\n",
    "        name_file_dict = {}\n",
    "        for file_name in model_file_names:\n",
    "            name = file_name.replace(\".pdparams\",\"\")\n",
    "            name_file_dict[name] = os.path.join(model_dir, file_name)\n",
    "\n",
    "        # 加载每层参数\n",
    "        for layer in self.model.layers: # 遍历所有层\n",
    "            if isinstance(layer.params, dict):\n",
    "                name = layer.name\n",
    "                file_path = name_file_dict[name]\n",
    "                layer.params = paddle.load(file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.2.6 模型训练\n",
    "基于`RunnerV2_1`，使用训练集和验证集进行模型训练，共训练2000个epoch。评价指标为第章介绍的`accuracy`。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.20000\n",
      "[Train] epoch: 0/1000, loss: 0.7368471026420593\n",
      "[Evaluate] best accuracy performence has been updated: 0.20000 --> 0.21875\n",
      "[Evaluate] best accuracy performence has been updated: 0.21875 --> 0.22500\n",
      "[Evaluate] best accuracy performence has been updated: 0.22500 --> 0.23125\n",
      "[Evaluate] best accuracy performence has been updated: 0.23125 --> 0.23750\n",
      "[Evaluate] best accuracy performence has been updated: 0.23750 --> 0.24375\n",
      "[Evaluate] best accuracy performence has been updated: 0.24375 --> 0.26250\n",
      "[Evaluate] best accuracy performence has been updated: 0.26250 --> 0.28125\n",
      "[Evaluate] best accuracy performence has been updated: 0.28125 --> 0.31875\n",
      "[Evaluate] best accuracy performence has been updated: 0.31875 --> 0.34375\n",
      "[Evaluate] best accuracy performence has been updated: 0.34375 --> 0.38750\n",
      "[Evaluate] best accuracy performence has been updated: 0.38750 --> 0.41250\n",
      "[Evaluate] best accuracy performence has been updated: 0.41250 --> 0.43750\n",
      "[Evaluate] best accuracy performence has been updated: 0.43750 --> 0.44375\n",
      "[Evaluate] best accuracy performence has been updated: 0.44375 --> 0.45000\n",
      "[Evaluate] best accuracy performence has been updated: 0.45000 --> 0.47500\n",
      "[Evaluate] best accuracy performence has been updated: 0.47500 --> 0.53125\n",
      "[Evaluate] best accuracy performence has been updated: 0.53125 --> 0.59375\n",
      "[Evaluate] best accuracy performence has been updated: 0.59375 --> 0.65625\n",
      "[Evaluate] best accuracy performence has been updated: 0.65625 --> 0.68125\n",
      "[Evaluate] best accuracy performence has been updated: 0.68125 --> 0.69375\n",
      "[Evaluate] best accuracy performence has been updated: 0.69375 --> 0.71250\n",
      "[Evaluate] best accuracy performence has been updated: 0.71250 --> 0.73125\n",
      "[Evaluate] best accuracy performence has been updated: 0.73125 --> 0.74375\n",
      "[Evaluate] best accuracy performence has been updated: 0.74375 --> 0.75000\n",
      "[Evaluate] best accuracy performence has been updated: 0.75000 --> 0.76250\n",
      "[Train] epoch: 50/1000, loss: 0.6607287526130676\n",
      "[Evaluate] best accuracy performence has been updated: 0.76250 --> 0.76875\n",
      "[Evaluate] best accuracy performence has been updated: 0.76875 --> 0.77500\n",
      "[Evaluate] best accuracy performence has been updated: 0.77500 --> 0.78125\n",
      "[Evaluate] best accuracy performence has been updated: 0.78125 --> 0.78750\n",
      "[Evaluate] best accuracy performence has been updated: 0.78750 --> 0.79375\n",
      "[Evaluate] best accuracy performence has been updated: 0.79375 --> 0.80000\n",
      "[Train] epoch: 100/1000, loss: 0.5855825543403625\n",
      "[Evaluate] best accuracy performence has been updated: 0.80000 --> 0.80625\n",
      "[Evaluate] best accuracy performence has been updated: 0.80625 --> 0.81250\n",
      "[Evaluate] best accuracy performence has been updated: 0.81250 --> 0.81875\n",
      "[Evaluate] best accuracy performence has been updated: 0.81875 --> 0.82500\n",
      "[Train] epoch: 150/1000, loss: 0.5149411559104919\n",
      "[Evaluate] best accuracy performence has been updated: 0.82500 --> 0.83125\n",
      "[Train] epoch: 200/1000, loss: 0.4710838794708252\n",
      "[Train] epoch: 250/1000, loss: 0.4474557042121887\n",
      "[Train] epoch: 300/1000, loss: 0.4346434772014618\n",
      "[Train] epoch: 350/1000, loss: 0.4275220036506653\n",
      "[Evaluate] best accuracy performence has been updated: 0.83125 --> 0.83750\n",
      "[Train] epoch: 400/1000, loss: 0.4235161244869232\n",
      "[Evaluate] best accuracy performence has been updated: 0.83750 --> 0.84375\n",
      "[Train] epoch: 450/1000, loss: 0.4212511479854584\n",
      "[Evaluate] best accuracy performence has been updated: 0.84375 --> 0.85000\n",
      "[Train] epoch: 500/1000, loss: 0.4199603199958801\n",
      "[Train] epoch: 550/1000, loss: 0.41921162605285645\n",
      "[Train] epoch: 600/1000, loss: 0.41876325011253357\n",
      "[Evaluate] best accuracy performence has been updated: 0.85000 --> 0.85625\n",
      "[Train] epoch: 650/1000, loss: 0.41848087310791016\n",
      "[Train] epoch: 700/1000, loss: 0.41829052567481995\n",
      "[Train] epoch: 750/1000, loss: 0.4181515872478485\n",
      "[Train] epoch: 800/1000, loss: 0.41804176568984985\n",
      "[Train] epoch: 850/1000, loss: 0.41794881224632263\n",
      "[Train] epoch: 900/1000, loss: 0.41786614060401917\n",
      "[Train] epoch: 950/1000, loss: 0.41778990626335144\n"
     ]
    }
   ],
   "source": [
    "from nndl.metric import accuracy\r\n",
    "paddle.seed(123)\r\n",
    "epoch_num = 1000\r\n",
    "\r\n",
    "model_saved_dir = \"model\"\r\n",
    "\r\n",
    "# 输入层维度为2\r\n",
    "input_size = 2\r\n",
    "# 隐藏层维度为5\r\n",
    "hidden_size = 5\r\n",
    "# 输出层维度为1\r\n",
    "output_size = 1\r\n",
    "\r\n",
    "# 定义网络\r\n",
    "model = Model_MLP_L2(input_size=input_size, hidden_size=hidden_size, output_size=output_size)\r\n",
    "\r\n",
    "# 损失函数\r\n",
    "loss_fn = BinaryCrossEntropyLoss(model)\r\n",
    "\r\n",
    "# 优化器\r\n",
    "learning_rate = 0.2\r\n",
    "optimizer = BatchGD(learning_rate, model)\r\n",
    "\r\n",
    "# 评价方法\r\n",
    "metric = accuracy\r\n",
    "\r\n",
    "# 实例化RunnerV2_1类，并传入训练配置\r\n",
    "runner = RunnerV2_1(model, optimizer, metric, loss_fn)\r\n",
    "\r\n",
    "runner.train([X_train, y_train], [X_dev, y_dev], num_epochs=epoch_num, log_epochs=50, save_dir=model_saved_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "可视化观察训练集与验证集的损失函数变化情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAENCAYAAAAYIIIKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8HWX1+PHPuXvWZm3apktSmu50gbRAAQtKWRSKFOFbRC1aRPz+QMSNRf2qoLKI0soXvoogCgJlFQooBamAgEBTit03uqZruqRtkpu7nt8fdxpu04WkTe7Nct6v1yV3nueZmTN3yj135pl5RlQVY4wxpqVc6Q7AGGNM52KJwxhjTKtY4jDGGNMqljiMMca0iiUOY4wxrWKJwxhjTKtY4jDGGNMqljiMMca0SsoSh4icKyIrRGS1iNx4iPq7ReRD57VSRGqT6mJJdbNTFbMxxpiDSSruHBcRN7ASmARUA/OAy1R16WHaXwuMVdWvOdN1qprd0vUVFRVpWVnZMcdtjDHdyfz583eoavEntfOkIhhgPLBaVdcAiMgs4ELgkIkDuAz4ydGurKysjKqqqqOd3RhjuiURWd+Sdqk6VVUKbEyarnbKDiIiA4ByYG5ScUBEqkTkXRH5/GHmu8ppU1VTU9NWcRtjjGmmI3aOTwWeVtVYUtkAVa0EvgjMEJHjms+kqveraqWqVhYXf+KRljHGmKOUqsSxCeiXNN3XKTuUqcDjyQWqusn5uwZ4HRjb9iEaY4xpiVQljnlAhYiUi4iPRHI46OooERkK5AP/TirLFxG/874IOJXD940YY4xpZynpHFfVqIhcA8wB3MAfVXWJiNwCVKnq/iQyFZilB17qNQz4vYjESSS62w93NZYxxpj2l5LLcVOtsrJS7aoqY4xpHRGZ7/QnH1GqLsc1xnQC8XicHTt2UFtbSywW++QZTKfhdrvJy8ujqKgIl+vYeikscSTReJzoou24emXjLmnx/YbGdBnV1dWICGVlZXi9XkQk3SGZNqCqRCIRtm3bRnV1Nf379z+m5XXEy3HTJrpxD3sffIeGvy5MdyjGpEV9fT2lpaX4fD5LGl2IiODz+SgtLaW+vv6Yl2eJI4l3QD6uggDxxsZ0h2JM2hzraQzTcbXVvrV/Ic1FBenhJVYfSnckxhjTIVniaMY7oAiA8AcbP6GlMcZ0T5Y4mvGdNoB4bYjoqu3pDsUY00EsX74cETnmwVNffvllRIQdO3a0UWTpYYmjGe/QYqL/3kr0oz3pDsUY00IicsTXsT5moaKigi1btjBmzJi2CbiTs8TRjIggMS+hx1akOxRjTAtt2bKl6fXMM88A8MEHHzSVzZs375DzhcPhFi3f7XbTq1cvPB67gwEscRyS/8wyYpv3El60Jd2hGGNaoFevXk2vgoICAIqLi5vK9o+Y3atXL372s59x1VVXUVBQwKRJkwC46667GDVqFFlZWfTp04cvfelLbN/+8enq5qeq9k8/++yznHfeeWRmZjJo0CAee+yxVsf+1ltvcdpppxEIBCgoKOArX/kKO3fubKpfv349n//85yksLCQjI4NBgwYxc+bMpvqnn36a0aNHk5mZSX5+PqeccgqLFy9u/YfYCpY+DyFwziDCG7bR+P5H+I7vne5wjEmr2m//nciHW1O6Tu+YXuTNOK9dlv3rX/+aG2+8kffee6/p7ngRYcaMGZSXl7N582auv/56vvzlLzNnzpwjLuuGG27gjjvu4J577uF3v/sdV1xxBRMmTGjxqbGNGzdyzjnncMkll/D73/+eHTt2cPXVVzN16lReffVVAL7+9a/jdruZO3cuPXr04KOPPmpKLBs2bGDq1Kn8+te/ZvLkyQSDQebPn4/b7T76D6gFLHEcgntQAfEtDTAsn3hDGFemL90hGWPayOmnn87NN998QNl3v/vdpvfl5eXMnDmTCRMmsHPnTgoLCw+7rOuvv54pU6YA8Mtf/pJ77rmHN954o8WJ47e//S0lJSU88MADTafB/vSnP3HyySfz/vvvM378eNavX89Xv/pVRo8eDXDAsjdt2kQ8HufSSy+ld+/Ej9zhw4e3aN3HwhLHIYgI7uJcAKIbduIbakcdpvtqr1/+6TJ+/PiDyv7xj39wxx13sHz5cmpra4nH40DiNNGREkdyZ7nP56OoqIht27a1OJYlS5YwYcKEA/pOxo8fTyAQYMmSJYwfP57vfOc7XHPNNTz//POcccYZnH/++Zx66qkAjBs3jokTJzJkyBAmTZrEGWecwZQpUygtPeQDVtuM9XEchv/UxGW54UWb0x2KMaYNZWVlHTC9evVqzj//fIYMGcITTzxBVVUVTz31FPDJnec+34FnI0SkKem0lW984xusXbuW6dOns2HDBiZNmsSVV14JgMfjYe7cubzyyiuMHTuWWbNmUVFR0XSaq71Y4jiMwGcGEl28k/jeejTWtv8QjDEdx3vvvUckEmHGjBlMmDCBIUOGsHVravp0RowYwTvvvEM0Gm0qe//992lsbGTkyJFNZX379uXKK6/k0Ucf5b777uOPf/wjoVBidAsR4eSTT+ZHP/oRb7/9NuPHj+dPf/pTu8ZtieMwXHkZsCVE6Mk14LLB3ozpqgYPHkw8Hufuu+9m7dq1PPPMM9x2220pWfd1113Htm3buPLKK1myZAlvvPEGX/3qVznrrLMYN24cAFdffTUvv/wyH330EYsXL+a5557juOOOw+/38/rrr/PLX/6S999/nw0bNvDKK6+wdOnSdu/nsMRxBP5PlRN+ZR3xTXvTHYoxpp2MGzeO3/zmN8ycOZPhw4dzzz33cPfdd6dk3X379mXOnDmsWrWKE088kYsuuojKykpmzZrV1CYWi3HttdcycuRIJk6cSCwW44UXXgAgPz+fN998kwsuuICKigquuuoqpk+fzg033NCucdsTAI8gsqyGHZ97hKybTibnylNsmGnT5S1btoxhw4alOwzTjo60j1v6BEA74jgCz9AivGN6gidGfEddusMxxpgOwRLHEYgI3sE90Vic8KqWX2JnjDFdmSWOTxCYPJTYyloiKy1xGGMMpDBxiMi5IrJCRFaLyI2HqL9bRD50XitFpDapbpqIrHJe01IVM4DvpL7EVu0FjRPb05DKVRtjTIeUkjvHRcQN3AtMAqqBeSIyW1WX7m+jqtcntb8WGOu8LwB+AlQCCsx35t2dktjdLjyl+cRrQ8R3NeDukZmK1RpjTIeVqiOO8cBqVV2jqmFgFnDhEdpfBjzuvD8HeFVVdznJ4lXg3HaNtpmMC4dR/523iC6whzsZY0yqEkcpkPws1mqn7CAiMgAoB+a2dt724j+zDFdxJsFZi9FoLJWrNsaYDqcjdo5PBZ5W1VZ9Q4vIVSJSJSJVNTU1bRqQeNxk/NcIXKcUEFqwoU2XbYwxnU2qEscmoF/SdF+n7FCm8vFpqhbPq6r3q2qlqlbuf2hLW8q4dCS6s5HwEhv00BjTvaUqccwDKkSkXER8JJLD7OaNRGQokA/8O6l4DnC2iOSLSD5wtlOWUr5T+xFdUetcXRVM9eqNMZ3AFVdcwVlnnZXuMNpdShKHqkaBa0h84S8DnlTVJSJyi4hMTmo6FZilSeOgqOou4FYSyWcecItTllLicuEb2BOAyDI76jCmo7niiisQkcSNu14vRUVFnHbaadx5553U19enO7wuJWUPclLVvwF/a1b2P82mf3qYef8I/LHdgmuhjC+MoO7ZD0CFwMnHpTscY0wzp59+Ok8++STxeJydO3fy1ltvcdttt/Hggw/y5ptvUlJSku4Qu4SO2DneYXkr+xD7YAfhV6rpioNDGtPZ+Xw+evXqRZ8+fTj++OP55je/yb///W9qamq48cYD7zu+5557GDp0KIFAgIqKCn7xi180PRfjhz/8IUOGDDlo+d/85jc57bTTWhyPqnLXXXcxcOBAfD4fxx13HDNmzDigzfPPP8/YsWPJzMwkLy+P8ePHs2DBAgAikQjf+c536Nu3L36/n969ezN16tTWfixtzh4d2woiQmDicez94Vxiv9yNZ2BBukMyJiXq/77woDJvWRG+YX3QaIyGV5ccXD+oBF9FCfHGCMF/Ljuo3jekN96BxcTrQgT/teKAuqzzRrVZ7KWlpVx++eU8/PDDPPjgg7hcLn7605/y0EMPMWPGDMaMGcOyZcu4+uqraWxs5NZbb2XatGn88pe/5L333uOkk04CIBQK8cQTT3D77be3eN333XcfP/7xj5k5cyZnnnkmr732Gt/+9rfJyclh+vTpbN26lUsuuYSf//znXHLJJTQ2NrJgwYKmR8nec889PPnkk/zlL39h4MCBbNu2jbfffrvNPpujZYmjlTK+PJq6e9+l4blF5Fz/KRtq3ZhOYMSIEezdu5cdO3aQnZ3NnXfeybPPPsu55ybuJS4vL+fnP/853/rWt7j11lsZPHgwJ510Eg8//HBT4njhhRcIBoNceumlLV7v7bffzrXXXstVV10FQEVFBStWrOAXv/gF06dPZ8uWLUQiES699FLKysoADhjyfP369QwePJiJEyciIvTv37/pAU/pZImjlTz9euCfPAjyXcR21eEpzEl3SMa0uyMdAYjHfcR6V8B75Ppsf5seYRzK/lPLIsKSJUsIBoNcfPHFB/zwi8ViNDY2UlNTQ3FxMdOmTePHP/4xM2bMwOv18vDDDzN58mTy8vJatM69e/dSXV3Npz71qQPKJ06cyMyZM2loaGDUqFGcc845jBw5kkmTJnHGGWcwZcoU+vVL3IHw1a9+lUmTJjFo0CAmTZrEpEmTuOCCCw561nmqWR/HUfCfXIbG4oT+tSbdoRhjWmDJkiX06NGDwsJC4vE4AE899RQffvhh02vRokWsWrWKgoLEKeipU6eyb98+XnrpJWpqanj55ZeZNq1tx1h1u938/e9/Z+7cuYwbN45nnnmGwYMH8+KLLwIwZswY1q5dy1133YXP5+O6665jzJgx7N2b3qeSWuI4ChkXjyC2opbolt3WSW5MB7dp0yYeffRRpkyZgsvlYsSIEQQCAdasWcOgQYMOerndbiDxWNYLLriARx55hMcff5yCggLOOeecFq83NzeXvn378uabbx5Q/sYbb1BeXk5mZmLAVBFh/Pjx3Hzzzbz55ptMnDiRhx56qKl9dnY2F110Eb/97W+pqqpi2bJlvPHGG23wyRw9O1V1FFzZfkS8SMBNbNNuPH2tk9yYjiAcDrN169aDLsft2bMnt912G5D4Ir755pu5+eabERHOOussotEoixYtYsGCBdxxxx1Ny/vKV77CJZdcwrJly7j88subkkpL3XTTTXz3u9+loqKCM844g7lz5/J///d/3HvvvQC88847vPbaa5x99tn07t2bVatWsXDhQqZPnw7Ar371K/r06cOYMWPIzMzk8ccfx+12M3jw4Db6xI6Sqna514knnqjtLfjqKt098x+675Gqdl+XMamydOnSdIdw1KZNm6YkHr2gbrdbCwoK9NRTT9U77rhD6+rqDmr/hz/8QUePHq1+v1/z8vJ0/Pjxet999x3QJhwOa3FxsQL64YcftiiGz3zmM03T8Xhc77zzTi0rK1OPx6Pl5eV69913N9UvXrxYzzvvPC0pKVGfz6f9+/fX733vexoKhVRV9Xe/+52ecMIJmpOTo1lZWVpZWanPPffc0X5EqnrkfQxUaQu+Y0W74KmWyspKraqqatd1aDzOtoEz8QwvpuhvX2rXdRmTKsuWLTvgqh7T9RxpH4vIfFWt/KRlWB/HURKXi4zLRxGa8xHRTXvSHY4xxqSMJY5jkPHlUWTccAINsxelOxRjjEkZSxzHwDe0GEFQjRCP2QOejDHdgyWOY+QZUIRkeAi9tTbdoRhjTEpY4jhGmRePROsjhOfbkwFN19AVL5gxCW21by1xHCN3UTa6MwwZQjwYTnc4xhwTr9dLMGgPKuuqgsEgXq/3mJdjiaMN+Eb1pfEvK2j8hw1BYjq3nj17smnTJhoaGuzIowtRVRoaGti0aRM9e/Y85uXZneNtIOP8Yez5xt8IPryQzAuGpjscY45abm4uAJs3byYSiaQ5GtOWvF4vJSUlTfv4WFjiaAPidZPx5eMJL91KbNs+3CU2Yq7pvHJzc9vky8V0XXaqqo0EPj+YwKWDaHj+4AfaGGNMV2KJo434Ti0nvjdMdPPudIdijDHtyhJHG3G5XLg8PlylmYSXbEt3OMYY024scbShwGcGIx4XjS8uTXcoxhjTblKWOETkXBFZISKrReTGw7S5VESWisgSEXksqTwmIh86r9mpirm1vEN6onsjRFbtQJ2njBljTFeTkquqRMQN3AtMAqqBeSIyW1WXJrWpAG4CTlXV3SKSfLFxUFXHpCLWYyEieIqLqHvwTcKXV+I/szzdIRljTJtL1RHHeGC1qq5R1TAwC7iwWZuvA/eq6m4AVd2eotjaVOZFw5BcPw1//jDdoRhjTLtIVeIoBTYmTVc7ZckGA4NF5G0ReVdEzk2qC4hIlVP++fYO9lhIhpfMH1RCHw/xehuCxBjT9XSkznEPUAGcAVwG/EFE8py6Ac5Tqb4IzBCR45rPLCJXOcmlqqamJlUxH5J3RAnuijyCs5elNQ5jjGkPqUocm4B+SdN9nbJk1cBsVY2o6lpgJYlEgqpucv6uAV4HxjZfgarer6qVqlpZXFzc9lvQCoEzBgHYiLnGmC4pVYljHlAhIuUi4gOmAs2vjnqOxNEGIlJE4tTVGhHJFxF/UvmpQIe+3tWdl4k2xCDHTbzWRho1xnQtKUkcqhoFrgHmAMuAJ1V1iYjcIiKTnWZzgJ0ishT4J/B9Vd0JDAOqROQ/TvntyVdjdVTe8mLc/XNo+GuHD9UYY1pFuuLQyZWVlVpVVZXWGGJ1jez+xl+hLk7R85enNRZjjGkJEZnv9CcfUUfqHO9S3NkBvAOKCb20mlhNfbrDMcaYNmOJox1lXDIcz4k9aXjaRsw1xnQdljjakWtALhlXjyS8YOMnNzbGmE7CEkc7cudnQUSRHDex6j3pDscYY9qEJY52JCJ4yopwD82n/qnF6Q7HGGPahCWOduYb0xfxuAjPr053KMYY0yYscbQzd3EOxABiRDfa6SpjTOdniaOdiQgZpwwm9OhKGp+1sauMMZ2fJY4U8I4owXN8T4JP213kxpjOzxJHimRcOQLpHyC2dV+6QzHGmGNiiSNF3GU98JxUQsNf7XSVMaZzs8SRIt4RfXDl+Ai/uSbdoRhjzDGxxJEi3r4FqCrxWJTYDhu7yhjTeVniSBHxe3DnZOAZVUjj8yvSHY4xxhw1Sxwp5Bvbn/jGeoLP2tVVxpjOyxJHCvkGleDJzSX06hp7MqAxptOyxJFigSlDcfXKpPHFlekOxRhjjooljhRTX4zM/xlH8Dm7LNcY0zlZ4kgxT98CxO0iumUP8fpwusMxxphWs8SRYu7iXHC58AzPJ/Ty6nSHY4wxrWaJI8XEJXjKCvGMLiJod5EbYzohSxxp4O1fiGR5CS/Zioai6Q7HGGNaJWWJQ0TOFZEVIrJaRG48TJtLRWSpiCwRkceSyqeJyCrnNS1VMbcXT998PPn5xBbtIDR3bbrDMcaYVvGkYiUi4gbuBSYB1cA8EZmtqkuT2lQANwGnqupuEenplBcAPwEqAQXmO/PuTkXs7UG8HjLOG0ptppfgs8sInFeR7pCMMabFUnXEMR5YraprVDUMzAIubNbm68C9+xOCqm53ys8BXlXVXU7dq8C5KYq73Wg0Rub1JxB6cy0ai6c7HGOMabFUJY5SYGPSdLVTlmwwMFhE3haRd0Xk3FbMi4hcJSJVIlJVU1PThqG3k7jiKsvCNSCb8Fsb0h2NMca0WEfqHPcAFcAZwGXAH0Qkr6Uzq+r9qlqpqpXFxcXtFGLbceUEcOVm4BlbTNAeKWuM6URSlTg2Af2Spvs6ZcmqgdmqGlHVtcBKEomkJfN2Sp4BhbgH59H4t5WoarrDMcaYFklV4pgHVIhIuYj4gKnA7GZtniNxtIGIFJE4dbUGmAOcLSL5IpIPnO2UdXqefgWIS5B8L5H5m9MdjjHGtEhKEoeqRoFrSHzhLwOeVNUlInKLiEx2ms0BdorIUuCfwPdVdaeq7gJuJZF85gG3OGWdnrs4F8nyIz38drrKGNNpSFc8RVJZWalVVVXpDqNFVJWd5zxCbP0eei6/BhFJd0jGmG5KROarauUntetInePdkoiQMWUY0Y92EV3WCa4GM8Z0ey1OHCJypoiUO+97i8ifReQhEenVfuF1fRpX4oXgu2igna4yxnQKrTniuA+IOe9/DXiBOHB/WwfVnYhLcOUE8J7Sm0ZLHMaYTqA1Q46UquoGEfGQuJt7ABAG7HKgY+TpV0BsSy3R6lqia3fjKc9Pd0jGGHNYrTni2CsiJcBEYKmq1jnl3rYPq3vx9isAsKHWjTGdQmsSxz0kLod9lMSAhQCnAsvbOqjuxpWbgatHBt7TSu10lTGmw2vxqSpVvUNE/grEVPUjp3gTcGW7RNbN+Mf0J76mnoZ3NhLbsg9375x0h2SMMYfUqstxVXXl/qQhImcCvVV1UbtE1s14B/Yk47xhoND4vB3EGWM6rtZcjvuGiJzqvL+BxNDoj4nIze0VXHfj6puN75wyuyzXGNOhteaIYyTwrvP+68CZwMnA1W0dVHcV/nADvosHEnp9HfFdDekOxxhjDqk1icMFqIgcR2KokqWquhGwa0fbiKdfIeIRXP1zaHxxZbrDMcaYQ2pN4ngL+F/gLuCvAE4S2dEOcXVLntJ8EMF3eqmdrjLGdFitSRxXALXAQuCnTtlQYGbbhtR9id+DuyQXz7ieNM75iHhdKN0hGWPMQVpzOe5O4OZmZS+1eUTdnKd/IbGtexCfi9DLq8n4woh0h2SMMQdozVVVXhH5mYisEZFG5+/PnAczmTbiG1RC9n+NR/weO11ljOmQWjNW1Z3AeBJXUa0nMVbVj4Fc4Pq2D617Er8HwUPgwqEEn1iMhqKIvzW7yRhj2ldr+jguASar6iuqukJVXwEuAi5tn9C6r+jWPXjOKEEjMUJz16Y7HGOMOUBrEsfhHk1nj6xrc4qGw3gqS+x0lTGmw2lN4ngKeEFEzhGRYSJyLvCcU27akLtnLvg8+M8to/G55Wgsnu6QjDGmSWsSxw+Af5AYGXc+idFy/wl8vx3i6tbE5cLTNx9XaSbxnQ2E39qQ7pCMMabJEXtdReTTzYped14CqFN2GjC3rQPr7rx9C4iuqcE1JJ/gs8vwTyxLd0jGGAN88lVVDx6mfH/S2J9ABn7SipxTWzMBN/CAqt7erP4K4FckhmoH+F9VfcCpiwH7R+HdoKqTP2l9nZ2nbz7uklz8E/rR+OwydMa5iFh3kjEm/Y6YOFS1vC1WIiJuEqe4JgHVwDwRma2qS5s1fUJVrznEIoKqOqYtYuksxO8l67OjkZ0Q/ONCIlWb8Y0rTXdYxhjTuudxHIPxwGpVXaOqYRJDsl+YonV3av6zB0Kuz66uMsZ0GKlKHKXAxqTpaqesuYtFZKGIPC0i/ZLKAyJSJSLvisjn2zXSDkRDUern/IeM6SMJPrMUVf3kmYwxpp2lKnG0xAtAmaqOAl4F/pxUN0BVK4EvAjOcUXkPICJXOcmlqqamJjURtzPxe3AX5uAZUUBs1S6ii7enOyRjjElZ4tgEJB9B9OXjTnAgMYiiqu4fDvYB4MSkuk3O3zUkruoa23wFqnq/qlaqamVxcXHbRp9GngGF4FKkVyYNj9lTeo0x6ZeqxDEPqBCRcmdQxKnA7OQGItI7aXIysMwpzxcRv/O+CDgVaN6p3mV5BxQBEJg6lOCjC9G43QxojEmvlCQOVY0C1wBzSCSEJ1V1iYjcIiL7L639logsEZH/AN8i8fwPgGFAlVP+T+D2Q1yN1WW5cgK4CrPxjC4itnGv3QxojEk76YodrpWVlVpVVZXuMNpMdNteUGX7oP8l4/JR5P/+gnSHZIzpgkRkvtOffEQdqXPcHIanJBdPrx4ELhpG8MklaCia7pCMMd2YJY5OIrppN/7J5WhtI41/X5XucIwx3Zgljk4iun0vsbo63Mfl0fCXhekOxxjTjVni6CS85YlLjANXjKTxxZXEa4NpjsgY011Z4ugk3HmZuAqycA/KhVCM4BNL0h2SMaabssTRiXgHFqPBEN5TS6l/8IN0h2OM6aYscXQi3vJiXDkBMqaOIDJvM5FF29IdkjGmG7LE0Ym4sgNkXVxJ5mVjwOe2ow5jTFpY4uhkRARXXoCMi4fR8MhCu6fDGJNyljg6GY3GqHvyfXxTjkN3BQk+tzzdIRljuhlLHJ2MeNy4CrJQjeAe0IMGO11ljEkxSxydkHdQCVoXIvPqsYReXUN03e50h2SM6UYscXRC3gGF4HXjHlMELqH+d11nQEdjTMdniaMTEo8bb3kxsZo9BKYMpf4PH6DBSLrDMsZ0E5Y4OinfyFIyzx5J1jfHobuCNMxanO6QjDHdhCWOTsrdIxNPSQ/8Z5bjGdmT+nveoys+W8UY0/FY4ujE4vUhGt/9iKxrKoks2Er4nY3pDskY0w1Y4ujkIiu24B5diOQFqL/nvXSHY4zpBixxdGKuLD+e0gKi63aQ9bWxBJ9ZRnRDbbrDMsZ0cZY4Ojnv0N5oMIx/6lAA6u5+N80RGWO6OkscnZynNB/J9hPbsYeMLx5Pw/3zie1sSHdYxpguzBJHJycuwT+iFFePDLK/PwFtiFD/v++nOyxjTBeWssQhIueKyAoRWS0iNx6i/goRqRGRD53XlUl100RklfOalqqYOwvf8FIyJlTgG1lC4ILB1N/zHvH6cLrDMsZ0USlJHCLiBu4FzgOGA5eJyPBDNH1CVcc4rweceQuAnwAnAeOBn4hIfiri7kxUlei2vWT9YALxnUEb/NAY025SdcQxHlitqmtUNQzMAi5s4bznAK+q6i5V3Q28CpzbTnF2WvGddTT87T+4inz4Tu/PvjvfRhttGBJjTNtLVeIoBZLvTqt2ypq7WEQWisjTItKvlfN2a67CbFxFOYQWbyLnp2cQ37TPBj80xrSLjtQ5/gJQpqqjSBxV/Lk1M4vIVSJSJSJVNTU17RJgRyYi+I8vRfc14h6Yi/9DwXYMAAAX5klEQVTT5ey77S3idaF0h2aM6WJSlTg2Af2Spvs6ZU1Udaeq7v+WewA4saXzOvPfr6qVqlpZXFzcZoF3Jp7+RUhOgPDianJ+fibx7fXU32NXWBlj2laqEsc8oEJEykXEB0wFZic3EJHeSZOTgWXO+znA2SKS73SKn+2UmWb2X5ob3xvEO6qEwPmD2Xfn28Rrg+kOzRjThaQkcahqFLiGxBf+MuBJVV0iIreIyGSn2bdEZImI/Af4FnCFM+8u4FYSyWcecItTZg7BW9GL7EvH48ryk3PrmWhtI/t+9U66wzLGdCHSFYfirqys1Kqq7t0xrHGFSJTd02cTfHopJcuvwVNmVzEbYw5PROarauUntetIneOmjagq9S99SPCd1fS4/SzEJez9wavpDssY00VY4uiCRARPaT7RdTsg00P2DacRfGopoX+tT3doxpguwBJHF+UfUQo+N6EF68n+/gTc/XLZc93f0Wgs3aEZYzo5SxxdlPi9+IaXEt2wEw2Gyb3rbCILttoAiMaYY2aJowvzDy8Fn4fwqq1kXDIC/2cr2PujufawJ2PMMbHE0YWJ30PWZ0cRGH8cIkLevZ8Fhdr/fomueDWdMSY1LHF0ce78LMQlaCiKu38eObeeSeilVQRnLU53aMaYTsoSRzcQ3xuk7pl5RFZvI/tbJ+E9qZTa/36JWPWedIdmjOmELHF0A5ITwJWbQeiDdRBXCv4yBSIxdk97Do3H0x2eMaaTscTRDYgIgZOOQ4MRQh+sxzOokB4zzyM0dy11M95Nd3jGmE7GEkc34S7OwTu0N+Hlm4ntqCPza2MJfH4oe296jXDVQYMNG2PMYVni6EYCJ5QhAS+RDTsSV1n94QLcvbLZdfGTxHbUpzs8Y0wnYYmjGxG/h6wLTyBwQhkA7qIsCp65lNi2OnZf9gwas/4OY8wns8TRzbgyfADEauuJ7Qniqywl777PEfrHGvb+8LU0R2eM6QwscXRDGovTMGcxwTdXoHEl62snkPmNE6m7423qH5if7vCMMR2cJY5uSNwuAuMGEt+xj/CijQDk3fNZ/OcOovbqF2n8+6o0R2iM6cgscXRT3oHFeMqLCS1YT3TbHsTrpuDJS/COKmHXJU8Snr853SEaYzooSxzdWMaEQbiyAwRfX068MYwrx0/hS5fjKs5i5zmPEFm4Nd0hGmM6IEsc3Zj4PGScOQzvwGLE6wHA3TuHote+gmR42fGZh4ks3pbmKI0xHY0ljm7OXZhNYNxAxO1CI4mHPHkGFlD0z2ngc7Pj03+2Iw9jzAEscRgA4nWN1P11PuGViSThGVSYSB5eNzWfeojQm+vSG6AxpsOwxGEAkEw/rh4ZNL6zmujmxIOevIOLKH5nOu5e2ew4+xGCzy1Lc5TGmI4gZYlDRM4VkRUislpEbjxCu4tFREWk0pkuE5GgiHzovH6Xqpi7E3EJmWcOw9Ujg4Z/LiW2sw4Az4A8it76Gt4xvdg15Qn23fmWPQTKmG4uJYlDRNzAvcB5wHDgMhEZfoh2OcB1wHvNqj5S1THO6+p2D7ibEp+HzEkjEK+HhjmLiO1pABJDkxS9No2MS0aw94Z/sPuyp4nXh9McrTEmXVJ1xDEeWK2qa1Q1DMwCLjxEu1uBO4DGFMVlmnFlB8g853jcvfNwZfo+Ls/ykT/rC+TefhbBJ5dQM+FBIit2pDFSY0y6pCpxlAIbk6arnbImInIC0E9VXzrE/OUiskBE3hCR0w+1AhG5SkSqRKSqpqamzQLvjtw9Msg8cxji9aCRaNORh4iQc8NpFL50ObHqvdSM/R3191fZqStjupkO0TkuIi7gN8B3D1G9BeivqmOB7wCPiUhu80aqer+qVqpqZXFxcfsG3I0E31lNw0v/IVazr6kscF4FJYu+ie/U/tR+40V2XTSL2NZ9R1iKMaYrSVXi2AT0S5ru65TtlwOMBF4XkXXAycBsEalU1ZCq7gRQ1fnAR8DglERtCIwdAF4P9S8vIrppd1O5u08uhXO+RO6vz6bx76vZNvR/qf99lT2K1phuIFWJYx5QISLlIuIDpgKz91eq6h5VLVLVMlUtA94FJqtqlYgUO53riMhAoAJYk6K4uz1XbgZZnxuNKydAw6uLCS3Z1HRqSlwucr4zgZ6Lvon3hN7UXv0iO05/iPCHW9IctTGmPaUkcahqFLgGmAMsA55U1SUicouITP6E2T8FLBSRD4GngatVdVf7RmySuTJ9ZH1uNJ5+hYQXV0M4ekC9d3ARRa9NI//Pnye6cic1J/yeXdP+SnTjnjRFbIxpT9IVOzYrKyu1qqoq3WF0OaqK1odxZfvReJz4vkbcPTIPaBOvDbLvtreom/kuiJB97XiyvzsBd0l2mqI2xrSUiMxX1cpPatchOsdN5yAiuLL9AIT/s5H65xcQXrHlgKuqXHkZ9LhjEiUrriXjkuHU3fUOW8tmUHvt34iur01X6MaYNmSJwxwV75DeuEtyaXxnNQ2vLCa+L3hAvWdAHgUPT6Hn8mvIvPx46n9fxbZBv2XX5c8Qemu9XcJrTCdmp6rMUVNVIsu30Dh/HcSVjNMq8A7seci20Y17qPvNv2l4aAG6J4Tn+J5kfXMcmV88HlePQGoDN8YcUktPVVniMMcsXh+i8b2P8I/pj7sgG43GwO1CRA7RNkzw8UXU3zePyIKt4HcTOH8wmV88nsBnK5CANw1bYIwBSxyWONIo+OYK4vsa8Z9YhqdXj0O2UVUi8zbR8OgigrMWE99ej/TwE7hwKBkXDMZ/9nG4cu1IxJhUssRhiSNtwiu3EvpgHRqM4C7JxTeqH57S/EMegQBoNEZo7lqCjy0iOHsFursRvC78nxpA4PzB+M8aiGd4MeKyLjlj2pMlDkscaaXRGJGVWwkt3oTWh/CPK8c/sm+L5gv/u5rGF1fS+OJKoksT4465ijLxfWoA/jPK8E8cgGdET8RticSYtmSJwxJHh6CxOJE12/H0ysOVEyC6eTeR9TvxVZTgKsw+7FHIftF1uwm9vo7w6+sIvbGe2LrEJb2S6cV7Qm+8lX3wjeuDt7IPnkEFdlRizDGwxGGJo0MKLdlEaP5aiCmu3Aw85UV4Bxbjzstq0fzRdbsJ/2sD4arNROZtIrxgKzQm7mSXLC+eYcV4hhfjHe78HVGMe0CeHZ0Y0wKWOCxxdFgaihBZv5PImhpiW2uRrADZX6hERIjtrseVm9HiL3qNxoguqUkkkkXbiC6tIbKkhvjmpNF6vS7cZXl4yvNxlyf+egYm3rv75uIqzrLEYgwtTxyeVARjTDLxe/EN7oVvcC/iDWHidY2ICBpXGv6+EI3F8fTqgbukB+6SXNyFOYjn0F/s4nHjHd0L7+heB5THa4NElu0gumQ70dW7iK3ZTXRtLZH5m4nvPPBmRdyCqyQbd+9s3H1ycPXOSbzvnYOrMANXYWbib0HivWTYJcOme7PEYdLKlek74EmDgdOHEK3eRWxLLdHqxDDuvpF9CYwrRyMxIut24C7MxpWXccT+DFdeBv5T+uE/pd9BdfG9jcTW1RJdW0ts017iW+qIbd5HbMs+Yhv2EH5vE/GaejjMwbhkeBAnibjyA7hy/EiuH8nxJd4n/ZUcPy6nTjK9SIYXCXgSy3De43N/Yl+PMR2JJQ7TYYhL8PYrwNuvAIB4Y4TY9r24chL3c8R219P41spEY5fgysvElZeJf0Qp7qKcxI2HcUV8R/5n7coN4BrVC++oXodto5EY8e31xHcFie9sIL7T+bsreOD7XUFi1XuJ7wuh+8LovhAajB52uYfecBIJJDmpBJzE4neD1414XQf9Fd/h6/C6nXpX4mjN7ULcAm4XuATckjg95xZwJb136uVQ7w/RVpxl4XaBJPaLiDS9p9l7OUw5ktj/hyr/5GVa0k01Sxymw3IFvLj6FzZNu4tyyLroROK76ojtqk+8tu9DhyQeHhXduIvg68sRvwfJ8uPK9CNZPvyj+uHKDhBvCKONkcSXst97xH4N8bpxl+biLj3oYZOfSKMxtC5MfH8i2RcmvjeENkTQxigajEBjFA0m3uv+905d8ntCsUQSa4xCJIZG4h//DSfqiMQP+tvtCK1ORjgJR5Ln3Z+EDlMmLWzXtP4WtjuaWJoScbMyz4hi8n93wTF/pEdiicN0GuIS3HmZuPMy8Q48uN5VkIX/xDLidY1oQ5h4fQit2dt0/0hkzXZC89Z+PIPHjQQ8ZH1uDK5MH5F1O4hu2p34he51I87LW9ELcbuI7QmiwXCzX9yupiMijcUT/zN73EheBq68jFR8LAdRVYjF0XBSMokrxJzyw7wnps50HI3GE9sTVyQOxOLEo3GnXRyNaWKZTn08GgPVxCuuidN8qmgcBEE1DvF4UzmqqAJxkLgTs+yvB8VZTgwk5tR7cCqd5aPQqBAFRCEgiWUmt6mPQzgOHkGzXLD/YqD9pyFrIxBW8An08CS+fJtOUSq6PQwRhUwXkn9w31a8OphYfq4HV5HfmfnjhcQ/qodIHCn0IcX+RKmTAwBii/dAVJE+Gbh6+j+e1WkQfX8nKLjKsnD1Chw4c1yJ/ms7AK6hubh6ZSTqNrfyiPcoWOIwXYa7RybuUZmHrfcOKMSVHUAbI2gogoaiiSMQnxuA+L4g0epdzq/22MfzDU6c0gov3URkebOnG7qE3GmnAdD4zioiq7cn/UIUJOAl59LxAAT/tSLRb5NU78ryk/W50Yn6t1YS27b3wMX3yCDzrBEANMxdSmxX/cdfmoC7MJvMTw8HoH7OIuJ7kjr+VfH06kHGxKEA1D3/AVofSvpyBW//QjI+NQSAfY+/i4YiB/TteCtKCExMPKl575/+dVC/j29YHwInH4dG4+x75O2DPnPfqH4ETiwj3him7vH3Dqr3n1iGf1Q/4vuC1D198JWQgZOPwzesD7FdddQ/vwDnw/u4/vTB+AaVEN22h4a/LUyqScj49DC8A4qIVO8i+OqSg5af+fVKPH3yiaytIfj68oPr//sUPD1zCa/cSuPbqw6qz7nuNNz5WYSXbqaxyvlRIs5/BHK/dyaubD+hxdWEF278+MjBadjjxrMQv4fQwo2EV2xpmi+xCKHHj85GXC5CH24gsrbmgPnF7SLvf84FILRgfdOjnTNvGHVQnG3NLsc15hBUFaKJX+v7O+9jexrQulDiuer7f7Gr4htUAkBkw07iu+oTv9T3/+L1uBLPbQfCK7YQ21n38a9uQPweAuMSh0+hDzcQq234OAgBV6afwLhyABoXrEf3NR4Qpys3A/+Y/on6+evQhnDTvECiD8g54mqcvw4NRw843eLKz8JXUdK0/sRRU6JOAFdBNt4BidOFoUXVHy/bOT3iLszC0ysPjSuRVVsPPG2C4C7Iwl2YjcbiRNfvPPC0C4l+KnePDDQa+zhpJrVx5WTgyvKjkRix2noEObA+y4/4vWg0Rrwu1LRccfKLBHyI1504ggpFDvhSBxCfB3G7Etsd3b/tSW2cwTp1//7cP18X7Vex+zgscRhjTKvYEwCNMca0C0scxhhjWsUShzHGmFZJWeIQkXNFZIWIrBaRG4/Q7mIRURGpTCq7yZlvhYick5qIjTHGHEpKLscVETdwLzAJqAbmichsVV3arF0OcB3wXlLZcGAqMALoA/xDRAaragxjjDEpl6ojjvHAalVdo6phYBZw4SHa3QrcASRfc3ghMEtVQ6q6FljtLM8YY0wapCpxlAIbk6arnbImInIC0E9VX2rtvMYYY1KnQ3SOi4gL+A3w3WNYxlUiUiUiVTU1NW0XnDHGmAOkasiRTUDy+NZ9nbL9coCRwOvOHZm9gNkiMrkF8wKgqvcD9wOISI2IrD+GeIuAHccwf2dk29z1dbftBdvm1hrQkkYpuXNcRDzASuAzJL705wFfVNWDB49JtH8d+J6qVonICOAxEv0afYDXgIr27BwXkaqW3D3Zldg2d33dbXvBtrm9pOSIQ1WjInINMAdwA39U1SUicgtQpaqzjzDvEhF5ElhKYhzM/2dXVBljTPqkbHRcVf0b8LdmZf9zmLZnNJv+BfCLdgvOGGNMi3WIzvEO6P50B5AGts1dX3fbXrBtbhddcnRcY4wx7ceOOIwxxrSKJY4kLR1Pq7MRkX4i8k8RWSoiS0TkOqe8QEReFZFVzt98p1xE5LfO57DQuTmzUxIRt4gsEJEXnelyEXnP2bYnRMTnlPud6dVOfVk64z5aIpInIk+LyHIRWSYip3T1/Swi1zv/rheLyOMiEuhq+1lE/igi20VkcVJZq/eriExz2q8SkWlHG48lDkfSeFrnAcOBy5xxsrqCKPBdVR0OnAz8P2fbbgReU9UKEpc570+W5wEVzusq4P9SH3KbuQ5YljR9B3C3qg4CdgPTnfLpwG6n/G6nXWc0E3hZVYcCo0lse5fdzyJSCnwLqFTVkSSu2pxK19vPfwLObVbWqv0qIgXAT4CTSNze8JP9yabVVNVeiX6eU4A5SdM3ATelO6522tbnSQw4uQLo7ZT1BlY4738PXJbUvqldZ3qRuFn0NeDTwIskHvy5A/A03+ckLhU/xXnvcdpJurehldvbA1jbPO6uvJ/5eEiiAme/vQic0xX3M1AGLD7a/QpcBvw+qfyAdq152RHHx7rFmFjOoflYEiMQl6jqFqdqK1DivO8qn8UM4AdA3JkuBGpVNepMJ29X0zY79Xuc9p1JOVADPOScnntARLLowvtZVTcBdwEbgC0k9tt8uvZ+3q+1+7XN9rcljm5ERLKBZ4Bvq+re5DpN/ATpMpfYicj5wHZVnZ/uWFLIA5wA/J+qjgXq+fj0BdAl93M+iRG0y0mMLJHFwad0urxU71dLHB9r0ZhYnZWIeEkkjUdV9VmneJuI9HbqewPbnfKu8FmcCkwWkXUkhvH/NInz/3nOEDhw4HY1bbNT3wPYmcqA20A1UK2q+59n8zSJRNKV9/NZwFpVrVHVCPAsiX3flffzfq3dr222vy1xfGweUOFcjeEj0cF22KFQOhMREeBBYJmq/iapajaw/8qKaST6PvaXf8W5OuNkYE/SIXGnoKo3qWpfVS0jsS/nqurlwD+BLzjNmm/z/s/iC077TvXLXFW3AhtFZIhT9BkSQ/V02f1M4hTVySKS6fw737/NXXY/J2ntfp0DnC0i+c6R2tlOWeulu8OnI72Az5IYjPEj4IfpjqcNt+s0EoexC4EPnddnSZzbfQ1YBfwDKHDaC4krzD4CFpG4YiXt23EM238G8KLzfiDwPokHgj0F+J3ygDO92qkfmO64j3JbxwBVzr5+Dsjv6vsZ+BmwHFgMPAL4u9p+Bh4n0YcTIXFkOf1o9ivwNWfbVwNfPdp47M5xY4wxrWKnqowxxrSKJQ5jjDGtYonDGGNMq1jiMMYY0yqWOIwxxrSKJQ5jOigRKRMRTbqRzZgOwRKHMcaYVrHEYYwxplUscRjTCiLSR0SeEZEaEVkrIt9yyn/qPEDpCRHZJyIfiMjopPmGicjrIlLrPHRoclJdhoj8WkTWi8geEXlLRDKSVnu5iGwQkR0i8sMUbq4xh2SJw5gWEhEX8ALwHxLDUX8G+LaInOM0uZDEcBYFwGPAcyLidQaYfAF4BegJXAs8mjSm1F3AicAEZ97koeAhMWTMEGd9/yMiw9ptI41pARtyxJgWEpGTgKdUtX9S2U3AYGA9cK6qnuyUu0iMPHqp0/QpoI+qxp36x0k8YOcWEsOfn6yq/2m2vjISD2bqp6rVTtn7wG9UdVY7baYxn8iu1jCm5QYAfUSkNqnMDfyLROJoekiOqsZFpJrEMyIANu5PGo71JI5aikgMvPfREda7Nel9A5B91FtgTBuwU1XGtNxGEs9+yEt65ajqZ536pmcdOEccfYHNzqufU7ZffxJHJDuARuC4lGyBMW3AEocxLfc+sE9EbnA6tN0iMlJExjn1J4rIFOe+i28DIeBdEo/pbQB+4PR5nAFcAMxyjkL+CPzG6Xh3i8gpIuJP+dYZ00KWOIxpIVWNAeeTeObFWhJHCw+QeIocJB6k81/AbuDLwBRVjahqmESiOM+Z5z7gK6q63JnveySemzAP2AXcgf2/aTow6xw3pg2IyE+BQar6pXTHYkx7s181xhhjWsUShzHGmFaxU1XGGGNaxY44jDHGtIolDmOMMa1iicMYY0yrWOIwxhjTKpY4jDHGtIolDmOMMa3y/wFL6MXtGu9e3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 打印训练集和验证集的损失\r\n",
    "plt.figure()\r\n",
    "plt.plot(range(epoch_num), runner.train_loss, color=\"#e4007f\", label=\"Train loss\")\r\n",
    "plt.plot(range(epoch_num), runner.dev_loss, color=\"#f19ec2\", linestyle='--', label=\"Dev loss\")\r\n",
    "plt.xlabel(\"epoch\", fontsize='large')\r\n",
    "plt.ylabel(\"loss\", fontsize='large')\r\n",
    "plt.legend(fontsize='x-large')\r\n",
    "plt.savefig('fw-loss2.pdf')\r\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.2.7 性能评价\n",
    "\n",
    "使用测试集对训练中的最优模型进行评价，观察模型的评价指标。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Test] score/loss: 0.7600/0.4912\n"
     ]
    }
   ],
   "source": [
    "# 加载训练好的模型\n",
    "runner.load_model(model_saved_dir)\n",
    "# 在测试集上对模型进行评价\n",
    "score, loss = runner.evaluate([X_test, y_test])\n",
    "\n",
    "print(\"[Test] score/loss: {:.4f}/{:.4f}\".format(score, loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从结果来看，模型在测试集上取得了较高的准确率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "下面对结果进行可视化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f09b8085d10>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsnXd8XNWZsJ9z750ujbpc5CLZxr2CcQFjiqkxEDYhIZsQNr19YVPY9GxINiTZkCy7yW6y6W2zSyohhFADGHAAG5vibsvdsnqdXu695/vjnhmNZDvYIFsGn+f3kzSaueXcOzPve85bhZQSjUaj0WiM0R6ARqPRaE4PtELQaDQaDaAVgkaj0WgUWiFoNBqNBtAKQaPRaDQKrRA0Go1GA2iFoNFoNBqFVggajUajAbRC0Gg0Go3CGu0BnAi1tbWysbFxtIeh0Wg0ryo2btzYLaWse6ntXlUKobGxkQ0bNoz2MDQajeZVhRDiwPFsp01GGo1GowG0QtBoNBqNQisEjUaj0QBaIWg0Go1GoRWCRqPRaIBXWZTRy6F/+wGe/+IvaH1kI04mh3S9hkBCgJTeXyh9LJDSRQgDCSBdhBCA2hCpHkv1n0Co56T6z3telmw1lOFHOvZj75jeI29cqDNIMXgsqXYoPj7G9b16r1sW9x/Z61ZjfonrHrxWfd1n3nUPHeMrv25R3PdvXTeGwAz4qFs6i0VffAd1S2Ye5Uwjz2taIfRu3su9yz+Mk8qO9lA0Go3mhHBSWQ4/8Cxtj73ApffcRsNli0/6OV/TJqOn/9+3tDLQaDSvatxsnrXv/uYpOddrWiF0Pb1ttIeg0Wg0r5hUSxeu7Zz084yaQhBCBIUQ64UQLwohtgohvjTS55COO9KH1Gg0mlGh4GM4mYymDyELXCKlTAghfMBaIcT9UspnRnFMGo1Gc8YyagpBeuouof71qZ+TrwI1Go1Gc1RG1YcghDCFEC8AncDDUsp1ozkejUajOZMZVYUgpXSklAuBCcASIcTc4dsIId4nhNgghNjQ1dV16gep0Wg0ZwinRZSRlLIfeAy48iiv/UBKuVhKubiu7iXLeWs0Go3mZTKaUUZ1QohK9TgEXAbsGK3xaDQazelMIdP6ZDKaUUbjgJ8LIUw8xfQbKeW9ozgejUajOaMZzSijTcCi0Tq/RqPRaIZyWvgQNBqNRjP6aIWg0Wg0GkArBI1Go9EotELQaDQaDaAVgkaj0WgUWiFoNBqNBtAKQaPRaDQKrRA0Go1GA2iFoNFoNBqFVggajUajAbRC0Gg0Go1CKwSNRqPRAFohaDQajUahFYJGo9FoAK0QNBqNRqPQCkFzRuMASf010LwKkFKe9HPob4LmjGYbQdZQNtrD0GhOC0azhaZGMypIoAcTB8FeAiQxaMGHD0kNtv5SaM5Y9Gdfc8aRR/AY5eQRmEgMJE9ShgQuJEED+dEe4kmnBR8pDKaTHe2haE4jtMlIc8bhR7KaAaK4OAhcDASwiviIKgMXWE+YfgzWEhmx444EWwixlRAn3yqteTWhVwiaM5IwEpQ4FEhcIIozoufowGI3QXIIDhIgQZoy3BE9x4mQxOCvRHAQDGAigPuJYgBLSVI1wtevefWhFYLmjMSLLjI5hxTVOKyljB4sJozACmEzQdrx0YMFSA7iByT3EiWIZDaZUTHVBHGJ4rCXACAA6MdkPHnKtDLQoBWC5gzFBN5MnxKLcB39I3bsKhy2EVJrAYG3EhFIIIA7aj4KE1hGij4s+jABCCK5kETxPmhGj4L5bjTfi1HzIQghJgohHhNCbBNCbBVCfGS0xqI5MxHDHo/UF3ECec4joUSu9zU3kEhgOQkio2g2yiHox6QMlyocMhjEtStxVJEwip+IoYzmCsEGbpFSPieEKAc2CiEellJuG8UxaTQjQis+HLxZuQOU4RDDZC1lXE1s1MYlkCwgzQwyCKCZAJZ2LZ9yvDXj4KogFYkSSR77c3Gq3qFRmxpIKduklM+px3FgO9AwWuPRaEaSWmwuJc5SkiwlwRRyBJDEMYlhkMAYFTHsA2aTwcT78s8gqxzsmlNF4W6Xrgo6GqYggWR55VH3yQQjSHHyjUmnxVpRCNEILALWHeW19wkhNgghNnR1dZ3qoWnOMAYwyI6A8WgqOeqxqSfPesp4gZBaMUj+TAX3UEnHKVqgd2PyPKFTci7NsSmYhkpXBYW/AzVj6K8dx84F5+EYZnEbCTiGwc6zV5ySMY66QhBClAG/Bz4qpTxizSSl/IGUcrGUcnFdXd2pH6DmtGILQRIn8WO7ljI2Exyx44WRXEaMABIbAxsDA7iAOGOxR+w8f4udBNlJ8BSdTVPKcP9AKhJFAulwGe0TpyKBnQvPxxWCzUsvJV5Vx3Mrr6ZzfCOpSJTucZN4fsVq+mvHnZLxjmqUkRDCh6cM/ldKeddojkUzcnRhUYM94mI7h2AzIRwEC0iP2HGzCNqUzT+GSQaDWhxMJGPI4yIIvoRZpQOLKhz8Jds1E2AyOWpx8CHJIYt24/EnOdLIBh6lnCxGsXjfn6nARLKI9BmRjT1aFPwDBVLhciKpOABtk6bRtP05di5cgWMY9Ne2EauuRzg20jBACJLRKrYvvmg0hj56CkEIIYAfA9ullHeM1jg0I0seeIRyzifBxBESOofw0UyArBJsOwnSg0UAl/NIvmIDTwyTZ1Tsj8ArbfEUEYJIGsgxgMlY8swnc9T9JfAkZcwiwxy1TRrBs4QxkEwgTxKD2SoxbQMRurEYcxLn7BbQQJ5NhJQaghQGNdjU6LXCScdhMHKtbfJ0Gnc+j+0P0V/XwEDHYQZqxoBjs3npZQBIy/eSxzwV4aijuUI4H3g7sFkI8YJ67rNSyvtGcUyal8kABr1Y9GPiAtsJYiMI4jLuFQqgclwGsEirr5gNtGMxW0XKvBQH8RHFpfIYyVd12KwixiNEcVW+QASbJaR4ijKyCLrxUYlDA3kVTgoxDNrxkcYgj6BZFcqLK7MQeCUierGYRZoFSllMIH9KInvmkKEfkwP4i/fpEuI6+egVUD7ZpnZRln13Dy1FIos/AgNJJhzFn0vjWBY9YyZS3XmYgzMWeKah5Z4SwPKddu78UftsSCnXMro5GJoRpAMfG4gocSroxeJpfNSTp554UYi+HCpxuJIB/kAlqPnudLIsPA6zkQQ2EGEMec4neczt9uHHxcsXcPGymB+jvOQDKllLGbXYTCPLFHIkMdhYEqOTxmA3QUpTjJKYNGMyvWR1EThFYkDihb8GVRXXw/jpwKfNRa+A2e+PMeUNKfbfE8Z1vU97Qfmnw+WYdh4h4HDjdCbvepEdiy5ACsHmZZeCYYLreH9PQcTQy0FPFjQjwnSyhHF5UvUWEEiayFKDw+OUcwnxV3T8Q/gBQS15erFoxcc5R9nuBULMJEMGg+0EyGCQQ9CCn3VITBWH78Nz9h3CjwMcIADALNLYCPbjx0EM84RIBlRSF8A4bMaQpx0fhajy8eSYS5qHiSph4ZWqmK+UVws+ynGoOEWpSPNJM4UsPjzTmy5RceKYQZezPz2AFZE0XpvCDEhW/ncP2ZjB5t/WMbABTNfhcNNMGnZvYfeiFSTD5VR1tnrO4FIlYJ7eIvf0Hp3mVUdBCDoI8gj2EKAfkxQCHxKLE18WSmATIZrIsowkSQz2KAFeSgKDbYSI4DKWPIfwY5fkIO8hQH2JiHeA5wmRUiYeAWxV4ZkrSLCRCHbJbF4AlxIngstaItgIOkuUgcQzncUwkQgiOCQx6CrJWX6WCHXkWfE3VisvhbeSeWkEXp5BgZHy6RwPheiaV7IyPF1wc4LoFJsJlw6u8iavTpPpMbj/kan4gq347TzdYydR2dVGX914cF22LlvlbXyaK4FSRj3sVPPaYUAVSlvNAAAtBIhhYiC5m0p+SzWHeGnnWYEcgm5lA88hiGHQg4k9LMpoD34epZzHKQMkLxDiiaN0QavGZlWJ+coHrGaAGhwVHy6KyqofkyyC8iEViQSdWEX7fys+3BL1FsJV5exgPimuYYBLiBNA8hRh/kSUjFqtPEuY5widkIhux2IdYe6m8rQqN5FRhsJS9uHnEcpHZTwjjesKHr6plv0PhpFqYec68NtLJrHNXkAiWs32RRfgGgbbzr3Y28A0T2vT0LF49aguzWnPnBI7+aXEeVI5ZMFQJROSJzRLPYif9UQwlN+gH5OHqCCEy+vpL4rEKC69mOTUaiAPDBRF82BxuX7MI0ICfVDimpZKKUiaCTKOPGkEV5CgEof9+LGQGMAKkjyKoF1VMjWBaxk4YkY8FpsILvcTLa5NJF5Iaq1avwzHAdYTYWlJt+eCo74dH1KZtApO9dFUDRJ4iCjzSNNEjjye4mwmQA8WcQwCSDXqVw8FJ3Hh3maCEerObsd1IN4VonJMmr6rZuN2Gew450IwDHBdTxG8itEKQXNSqMemHIccljIheZyIUJhGFgvJ00SQCFygBpsLh82P69TM/wGiSCWwQ8pcM4jED2wlSBVOscx1ITxwESm2qBm7RJBFcBgfc0hTo0Y/lVzxaC7QjQ8LlzCSGCYDmFSX2OglsIYyEqpdZ2mEegiXFSUpdqWKqg0f+wjQSJZabB4kSnLIMSSbCbGZMLXkufwV+mdeDjZef4UkJglMmgkQxeZhKnBVJzoTuJcKJILlJGgquX+nI8OrjWYDIQwp8ecytE2dyaF1nfzx/gvZn5vMO1b+jmRdLU6ff1AJvMqVAWiFoDlJOEAPFhPJUY3NC4TZRZCZyqZ9vLPaSpzirF0CftyjJol1YCER1JOnC2tYQKr3OANsJsx8UkWFUJjZg2djv4cKUEplJpliqOhwJDCFLHNI40eyjeAR1ySARnI8QwRZfMZ7lMbgbqpYpuocPUQ5ZbjEMUlhAJJnKCOESwSXGCalakMgqcD+m5FTJ5N9BHhWrd4MJH1YPEglPlx8SLUy9EIwF5Ck8TRWBnLY40wgRDCb5nDTLKq6DhMdsGltOIu7vjxA88K5ICTfe/IfQBhgvprWPS/N6WOI1LymMIFzSXKQAC8QwQQyGPyKan5NlcopeGm6sIjgcAUxZpGh9xhzmAgu55HgUuJcQQwDWEaSBaQofOUtYAlJ5h5DyHcpl3c1DhJvpg6AkFTOyBXjghzARrCYFCE1E55H5qh5Dk3kmF0839A56EKSNJFjAIMefMXS1BklTDPKbzKLDBOKAtVbIUigieyoldKeRpYoNi7gqtVbBTavI1a8D6YK4Z2qjHmnE6W1giSQDnl5BTl/kERFNY5h0jp5OqlIBTvOXolrmjTPP8/byTBelf6B40ErBM1JYyo5FpHEVFFHLgIfLitJHHc336lkuYYBqnFYRPqYjWwmki/OQqtxeBP9TCFXjDLy42IjhpmRPFrw0YoPP5LFJLmCGFcRQwJ9mDRcnGH612Pco4JFtxBUDuyhOBy9TPEhfFhIgioXuuCCzSP4FVXcRwUCSTu+EvHu2dyXk6Qem3b8+EoUSghXheKeOl4kyE4CxDAYwFSrlsHVWxiXEC5dWEwixzkkMeGUFfE7XmTJD0DeFyBWWYtrGLQ0zcIVZlEJ7Jm3lJ4xE5Gm5ZmEXoNKoJTT653SvCZIYhBW8TfTybJJlXDwMoBPrGOYgVdryK8E5HArrQM8QRkrSODDEz71Jc5aB1hIiplk2KHyEobzAiFM4KqSPgUhXPoxOTzO5JobE9x/dw0pTHqmC/bvDpB0DbpUxFEFDgbwV8qowmbOsLPMIkMawSbCjCVHHTbbCVGGyxjydA6JVhpUKS6CnQSowaaBHEFcdhFEIJlMDgNJHo4rbsvGiwKreZl5CJ4jPIiJZKNKQCwE2xZG3I2FAF7HAFGl2hpOUVb2SzF8BLblJ+/3E04lONw0g1AyzvZFF9BbN57JO1+ge9wkkNJbDcBrXhEU0ApBM6K4wH1EOY8kDeTpxsIF5pImVhJC6j9OIdGJxaOUM4/0kCimAh34aMPPYfzUYPMIUa5QoaQAZ5eEp84uicnvwGIDYcK4xd5maygjgyCLgYvAsiSb2iJ86F0zKMwpH95VSWFW/BfKkQjOI04dDq346MWkFR/TyDKVHFkE48nzmArBXEwKH5KzyCKRrMMzVZgljveCr2ECWSpwyOMV3ysYXiSwgyARXGaSpbQAgoO3qqnF4THKWECaahzaKiy2mUEu6z2xWrHdmKyhXOUVCGzlMHaK4/Tagl5MnDKl1qIl65zjXQmeLAorgYL3JhWpIJIcoKVpBoF0imA6ScuUOUzcvZnu8Y2AZN/sxd7OxplnQNEKQTMiZBHEMejDIo9gFwECuPhwWV0yY5xFhuOp4JJRwqeZAC6CHQSZrGbJFvAcIQ4p5eI5YMNqTy8PYQo5IrjUY5PHW2mkMYpZxofwMaAigwqCrRUfNdj4kfRjIO3BqB4hJFIaFOaahZo1s0nxFOV4oaeSDAYpjGLI5XYlEk3182cVdTOD9JDVzlhyHC6agLxztODnMDCJPFczwL1UkFPivDaY55LMkXWJCqG6VzJAmzpeEzku+Uo3qyIuf3jnWCK41GIX78XfohqHqWTZQbAYOFoweRX+Wsgh0VWjzfBuZMnySsri/diWj/6asYRTMVqmzqFh73a2n70S17Q4MGORt7EwXjKbzrJsJk7oYN/+U9vPS54C3XrmqUDNSWE3AR6igo1KMHfi4yEqeIqyYnIXeAJm+OL7aJ/zB4hyD5UcUHH+WQT3UMlvqCKBQSM5XIQ6msBVs3oQdODjacrYh58uLP5IJTsJ8gjlrCfMX4mwj0BJDL83gnJcLifOKuIUMq6FgIYpGaT0REwgVGhxItWM2UMATnEMHtsIY+DVLnIARyW+TSTLTkLsIAjqucMq81oUf3sz8SuJUaWuMqeeGzsxyw9e2EGgzC2Ovg2L/fiLjXiKfgnh4xkzws//YyyLVibgPIfYUkmvdXSpF6gaKtgNYKGKpPLyQSRhXAw8/8Y80qRGqKnQSDC8G5ltWvTVjUcKQWvTTGzLZPuiFbiWn0NnzfdWBYZxQv6B5Uu28ImP/hLTOH2U4EihFYJmRJhNhsUkS8IrYQw5riD2kqLiScpoGWYJv4IY4SHZC55AXkaSMlyqcVhaDLksdRFSfK4Liy0EyWGwgyBJDFX9M0BeibeC4LBwiauqpa2qHEVQvXp4r9cwJ1ptY+cFE6Zmite4kxCFqJ+hBorB/3NK6Jtqu6WkOJ9EccQG0ESG1Qyo4NlCJJEgqu5BCoNJoSzXV/fy/je2EYm6TLw2TaDawQxINhHiKcqK3oti5U1TEgy7vOuzrSDhX3+9l9t/vZcFM44MVzX8kjc800bdudkhzw9gksVgAjnGYJPEpJo8PiRzyfAG+oeYANMIto5gk6HjQULx0+IYJoloFQCtk2dgmxY7Fp7Poalzsf1BuhqmqLpCSgkclyKQzJ29hyWLt7Dq4vUE/Dled8VTLFm8hdqavpN1WaccbTLSjAgCqFDhmkLNiEPIl3R4eoXnfNhQzA1A7TuBHLtKqoeGcZlSEs9+SEXlWMij5vzGsFSsEMVQzu5iBnNh1N7fKWoLGy/3YSkJ1lFGjcxjTXRZfFGcs+anWXltP3u3hnjsD5V0Hvbx7KNRhhoooFQpFITUVJVktoEIHSpZr/ReSYTKNBCcr7waTxEhhkkVDlU4/Msv9jPu/CyOktcrvtqL+U3Y8/sw+ZsFTxOhpeh+92b0ji246NpezrvSS16L95v89u/GYO4cXCFYEZfwWIe6c3L4yyXTb0yQ7TXIxQwyXSZhXC4hxlhV2Wkffp4hwm48p/Hw6q37CPAiIaaSfcnGQq+E4aahRLSaaKyX1sln4Zh+ymL9HJw+n7pDu+mcMBWAlqlzvI1fhpP4jdc9SuOkdnI5795du/oJDNPlRz99Pd09Va/8gk4DtELQjBidWERxOZ8EzQRUFdCj8wQRDqs6/SaeielXVOFD8nr6saBoU59Ijlb8pDBJIwgpYTqRHOPIs7ZYM6dU+HihptUTcrS3BEqe9Zygg3FAnkhpVjbyg/gJFmf1kgEsVl/azQdva8V1vDpls85JMm9ZkjV/rFAKQQ451uBYBAYQxGWaqvwaxyCKw0H8jCPPcpLFexXB5U30FdXU9cNCbB97Xw3Lv9XLhOVZzIAknxesf6CcvZ/yynSnMRg0Inn3pxuLZx+L4tqHvVek5Pc7a7iSWNGvM/8jMeZ9OI6T9aovTF6dZtqbUvRt83HPpWPxIxmLzV78RSe4gZencad6z66jn+cJk8IoRhs9ThkhvKq3I11Yr9Q0JABHGPSMm0R5rI8DMxZR27qfHQvPx7F8tDfOVG+PeAXZxIKv3P5OPvDuu5g/d7fyN7t88z9uZPvOpld6OacNWiFoRow5qmOYASwhdUw3o8RTAD41sy+UZDCBBaSKH8pLiJHHoBoHmyRdmMXZ6FaCbCpxJHuIIWdpDKT5xpo9/P3COaQSBVOKoB6bWaTZRqg4o5ZIfLgsJcl6IqSVSclG8sef1tLfbvJP323BtCT5nOAPP6rlR18eC0Aw5JJJGwxXDJbKfUhhspcAEdJsJ4QBQ7qvzSPDvGERVMNr6QBs7Ivw5GfK+M4zu8ilBb6g5JF7qmhPBUkh6MJiPinGYLORMDXYXDpxgAn/nOTR99XwYibMvGsS2Bjsx88s9V498LUaHnuqgg/9pAXT8FY17U8FWPPemiFjmqwUzF7l6BdILCQrSGDhZZE3Eyi+Dz1Y+JHMHqF2p6VKwABcYRCvqKaiv5u2SWeR9/nZufA8HJ+fjknTPAcxjFi0kG1bBAI5LMvBtg1M08V1X1tWd60QNCPG8K/G8LlYNyZPKBezoyzlpTNagaQDH9OUWagcSQaXDtVuclyJiplJhgQGBwgohTLouK6szTNxapbp89P4AhLHgbGTPDtL+8EAB/ETxmW8KpFdSKyyEexTcf+F5wEmkOOCsiQ+vySTEgTDkooam4BaSay6vpf1f6mgq21olJBNQUlImgnQrMxfhRagQSQLSeFXyrCUTQTJYrCEFCm8pL69BMi0G6z/Szl3/bCWj93RwoRpGZ6kori/jUEGg/NIEsUldchk2/ujPCvL2E+Ah9ZUAZJthNhCmBry+KWk50mLD5ktODmwgpKezT6yfUNHZQLjyLGbQDH0dCx5xirX+gIySATbSwxFVxAbElTwciiEvBrqqPGqOir6umifOJVUpIJofw/7Z51NdcchupRpCOPk1BWKlqf4319fwbPPzeYD776L8eO62Nk8+aSc6whOQS6EVgial00/JhbyqOGLbVj0Yg3JHajCYRLZonnGQxLBIa3KWrfgZ4fyCEwhy06C7CbAG+gfMv+38BzZewkUZ+KFmelVb+3jHZ9uJ5cRCAG/2byVYFjSvDnIR66YTj155pBhvVphlAZRHlAVTUEwgRyt+EhiEhlvs+nXZXzhc438w43tzFud5F9+s49PvXkaU2elefi3NRimxDAkTl4wjTTNhCiUzS51OBfyCmaT4QnKGU+uuELIAjPeF2fLs372Px9kNmn+RCVe6KrEcCS3vrMJELx35UwVilhQaV54rotgPinmkqEDH4/K6JDzA7hITHXferEwheQDF88g12Hwjo+1s/CiIwvmSeAZlaE9lzQtKgdEqr7WvZjsHuJREDxDmCocFpI+IWEz3D8Qq66jsrcTF0Hn+Eaifd3sm3UO0a42di5cjuPz09XQxNgxPcyYfoDH1x6tfdIr59avvK/4+PZ/v+mknGM00QpB87JZR4QwDhccpcDaNkL0Yg7pe2wC55DmAAFyeF/2CK5KBPNwgecI48fLwt1DgByCTYQI4FJPnnJcfHj9iqHUUOSJ3ju/XUesz+CDX271ys4AzzxUzu0fnIwLzCNNDxYt+FlNPw8Uq+9IojjEsFhEkllkSWDQio/N36rwVgPEWf/DSv7ww3ryld4+v/j3sZimZMXr+om2uTz0dFWx87Is+U3J43HkGE+ebQTJEKCBPCkEa80y7vrEYcSjkm3vn8Q9eM5KHy75IbVRIZ8bjL4qVZZLSRQrs9ZjD9u3sL8ggFvMw7BtQcteryT3U7dV8ot/HcvFJACvd3Qh5yOPKOaRTCTPgpKqVIUSJZS8K134sYrZE8dH4W45QmBIT9V1TphCRW8XHROmkgsE2TV/GbY/QO/4ycWZs+mDyy9dx/Klm1n71EIc99VfffRUoxWC5oTIIDikggz7MOnHpFnNz8eS46+UYSNUYxyv/LGJ5FxS1GGrxjNeCOMAxrBuy6IoQIO4bCJCQTxsJagimWwqcDmPJK34qFXHHNzfE3axgz5My2tkYgUliW6T2ekMzxFmB0Fsda5mgjgITFwcNe6CSWU3QRrJFmfvfiR+JGkcNmNh93vnHOj2YfokE6dl+Pld4wDYr1p+Dg1DpXg9bQRox69Kcxi0LDF552faeEu4DcOULL88xu1/3EM6YfDgB2p5Ll6GjRyiWgBqyJPBIKl6TriAD8lhfBzCxzJSXEKcB4kW75F3LS5hXKK4tCsnsAmsIs7TRGiz/SQxiKhVxFZ1z8CrwfQiISpxmFGyAqzA5e/o5y4qi6bABvKsJDGoJkyJdI40fQwqAQNTukggVlVPZW8HneObyAbCNM9bStf4Rsr6uuhumOLtYBj4fHne/vf3M2/ObiKRDEj42M13kk4HePCRZezeM/GI82mOzmvLI6I56WQRvECIDaq5vAs8S5hNyllah82AaiHpIEhgEMSlQs3Bw7hcSJwLSPA6Yowvzh4HzSqFhKzokLQvqMSmH4uD+HmMMnLK3BEd5r6+kDgXLR3g8PYA779oOvf9TzULzkkSwqUcm4ME6FC5Bs0qXr5B1QYqnC+HwGFoKGyBsdicT9wLYTckhil5/bu6ueHmLirKbeYOcaIOzUkIIlnNAJNUKlehHETrfj/BsKRxdoZAUOILSGYvTtG6P4Cb8IryzVb9kYHiWC0kIVxmkWEyWUygD4sdBNhPABuGpQJ6+60kzlKSdClfRlgJ/l9TxUH8GEjuoYJfUUU3FtcwUPI+eSuPs0kdUSwwppLzwqq+U0/JnNOKuLzp+VbC4+zicxKwS+z9seo6ALrGTqZ94lQkgt3zluCYJm2NM7D9AfrrVYawUM57yyYUzFBZkcRnOfh8DrN2v+1fAAAgAElEQVRm7KOpsZV4PIzm+NEKQXNCVOByNQP4kSo6x2szeTUDhJGcTZp6bFCvmkguJlFMXAqoWWNhRnp+MZnN+10oqdCOr2T14M2y+5RwkVBsbO8qpUDJMdZRxldun8h7LpvO4T1BvvOZCdy0aiZrKeNcUkwkW9xWAEuMBBdG4whDIAyJaXjz25X+OFU4dKh6TAXGrczQ32AgpaAuksexBb//Xh2fe2sT2YRgC2GOzE0QWHilLQpRWF7IpHefEp0WH7l6KnZe4NhemYJnHirnO59rYECaXEc/C8gwgImA4og68NGDxSF8tODHwatz1KlCfu+iiicow0JSh80yVYG0F4tyXOaQZiZpkhg0kikWovOiiGASOerI8wyRYrVagDgm64mwWZntCkgEM0lzDQO8jgHGkAchMfySiZenCdW6TL46heGXYHjnilXVFfdvmzTdUwLzl5GKRGmetxTbH2Sgdpy6jcKLGlJmooXzd/Hdf/8mZy/cheMIpPTunRDwr9+8iY7OoZFSmr/NqCoEIcRPhBCdQogtozkOzYlRKKNQKGWQxigKEhuv6mWhXo6NQe+wGJo1lBVLIhdyFcrUPLYXiwwGT6iCakeeedB57FFqkik4TGElCSIMlt12XYMVJKjDwa+eL5hY3IUu/W8QOC5IV9A0J40UwFU2GQSPUF4c58NEWPztPlZ8qB+QdMd9xXMffDJEVppMIjNsXN7qoGBy6cKiS4VkXkiC+aTJIZi+ME0gKGneFCLWZ7LgvCRR02YKuWKRuLNJEcFR66pCsTtR0pVtsAx3oU9BA3lWEedS4kwhx7X0M5kcO1WEVmGV1KlyITwHtouL10nuEH7a8DOWLKuIM40sAkkSgxb8bCHINoLk8VaIZ6s6TV5OSpIL/rOXt+9vYcW3ewFY/M8DvH1/C5fd2QVA22RPCXSNmUg2FGH33CXkA0Hi1fW0Nc7wbmGJEijlhU3T+fn/XoXtCEzTKzXiSi+YoLo6dsT2mr/NaPsQfgb8F/CLUR6H5gQ4pGbnK4nhR7KeCFm8hDEDL5fgLGW+2It/SNxJGq80RKFs9DaClOMQVzPfQtkGP26xRzJIAsr57HGkXV4Un/f6OVfgFDuPFXiKCD5lhz+LLLPI8BhlbHyuHPkcxXPt3hxhXEOWww+E6FKKoG+K4Nxrkry+PEO0yubiv+ujt9NHPiu452c15DJeQbsQLilMwoZL1hBU1OTp7QiUjNPrrmYB56ls6MLqI9Fv8p+fGc+ff1FDWYXD69/VQ1qaBJXiyuJlMx+Z7ieLRy/9v2BWWqJWBQUKyqUDi1YGk/ZSGKTwVnFnkWELIf5MBbZSnn34eFy5iItlIoBNhIjiYuFShmS8MrPllDJ1PutiVEomLMlglEmcrKBja5jHb66gp66BdKScPXMW09kwhUA6QWuTSiQ7rvwBwVPr5vO2Gx7Acbz372DLGJ57YSa28xpzKp+C6najqhCklE8IIRpHcwya4+cgPhKYHMYHSHYTpBabRapzGHhLzlklZaYL0S578LNR9fcy8KpyHlQhnnaJKgChCmB4jwvC3ztiQREMDeYsvDKBfHHGmmeowQYGC9FdQUy1uoFriHEIH0+qcEphwHlX9BPucXn4cHXxnPGQwfIP9hMucxEGlFW4/MMn29mzNcgff1JTHFMGg0ayLP9MP7OmpmhamuYtC+dg5wevMYZJHXlqsRlHjn0qmetgc4iDzd5s3eiHX97hJb49T4hOfCXO89J7U/pYFu9MBTb12DQTpANfUUiXspCMChstHGOQs8gSQPI8YXWnBDm8Wf+FxHiE8mLkUhU2lxPjT1QQwmWcOtchfPRhsTFWRusvTT678iD5pMAMStaunUOqs52dl5+PL5vm8JTZAOT9SkGdQMz99GkHAcH//eYyJk9qZ8XyTfzrN99BLn88nSI0pYz2CkHzKqILHzuLtYW8GkQt+DibFOOw/+a+k8jRpuzcbomgX0SSHQSHRBsNtb6XRuoMLQsRwsYCFRnkrT4mkaUTizwGFThDylsDNJIrKgOAYL3Ds92V4ArKq2zifSa7XwwRjw39auzdGuLdF8zg5+t2EAh649j8dIRP3TAVt8SnHSp3yU6SXPvWHgJlLsKCd36qjZ9+bRy22i6ISw8+lfXr3U/TcimrcBjo8c47oJQuoOo5lc7+S01ksrjdbNKqZHiw+J40kqMqmkckPHNYKRMbM1TsD6tCIah76rBarfxmkCWHYAuhonltJQlMJHllJnTw8lH+QjkZDNIY/IqhdX12E+DQ4z6+8ZFJNMkcF92aZM4FbfxyzrnkAyHyflUITxWaS7e1EaipwfAfX0e45j2T+OTnb6a3t4zYtm3cd/FyrQxeJqe9U1kI8T4hxAYhxIaurq7RHs4ZzTmkWFhSWqLgFJ5J9pj7SOAwFj7gXOVILRXs+wgcI/R0qLAbbiaaohKdYkWBL+hREUgZZSxJDjExeccpzHeTGOzBR/7NDlkp8Addbvx4O3//jx2Mn5Zl3vJ48ZxCSAIhh8o6m0BQkogZSAlzliZxnUHl0jAlyx92buG7D+3GX+4WU7Wv/1A3U+alCYe9yJtCHaVNhFUkleDv3tPNV+/cy1DBX7gPw1cDQxlPDgtJK35VisMz/xzCRxZYekcvs943NNFMWJKrH+igrt6bzRvqHmVK/EFAcRVXiG5qwUcGgwZyXEM/q4hjIenBwi26yI90qdt5wVMPRfnx3fX88yeu4he/vZo2tSoo9Q9Ix6Fv40ZShw4dcZ3HIpv109cfJdvdTXLvXg7uMJCui5M99udSc3ROe4UgpfyBlHKxlHJxXV3dS++gOSFOtKhAAgMbr3Ccg9cUZ/jrpcaJPkweJ8p+fMrUVMATdIVCaItVcltBHZjFkR3NkSxoxc8iUsdozyiK4ZwgqMXGh5dDMI48eaAViw1WhF/+1xgmTM3wjd/v5r//uYEn/lRF44wM6//iVTEd35RhxeoBXndjL2VRlw1ryvjAqul85oYpNG8MYfkGBfbhvQE+f2Mj6aSBaXlyLp8VdLSYJGMGqZRZNL940VEmY6dlmTo3xZVv7WHq7AwLzkswdW6aswIpdS1DM4yHKkjvcQoDC+jHIouBAJ7zhTg03eL5QJh7761l+luTVE7PU70wS/V5WWa8PYGvXPK2WzoYE8lzw9RuJpJDQkmqmVef6hoGWEKKy4hTj001Dheqvtj12CV9rgfHWrjOAo5tkEmaXPeZGv7j+3eRzgQpLT2d6+sjvmsXA5s3g+uS3LuX+K5dJPftQ0pJbU0fwcDRBXy8uZmedeu8fYGBF1+kc80aOtesASASSR1zX81QTnuFoDl5SLzEsX34WXOUpvEFcgi6SzJvz1GN6MHrMjagEqNa8PE4ZWwlRAcWzfh5ToUlbiJEDIOlJDmbFGaJz+ES4kqMCSaTw4dX2+cS4kwoCREdHLU3k32caLEwnseR8faV2Jh4hetmkaEah2aCbDZDfPfhXQgDLnp9PzMXpbnqbT18+76dbH4morqjwZJLYnz+Bwe46PX9LL9igNnnJvnWvbtpmpHmk9dNU76BwfE9+2gFiQETOw+5rCCXE3zncw0c3utlWpeufCJlDl+/by/febCZ2nF5clnBV+/cw3cfaqb+kjxhXGqKprjhqyWPKxlgARmuZoBq7GIp7fGNOVZcPUDaNXjqwSj33l1N7gaHC3/ZzTW/62TxF/qRLix+Q5xfNG/n8p9004oPA3g+GmL8lxO0Y9JIrliZqA672JpUAkaVQzP+YqOfIxACM+D5R+rG55gwNcNll24kFLK56a33MnZMN5blXZ9FivjOnaQOHgTASaeJ79hBpqMDgJs/8BtWX7n2yHMAZihEtrMTJ+lNKuxEAieRQGazZLu7ec+Nv+Mt1z9w1H1fCaHQkS1dTyanoq3zqPoQhBB3AhcBtUKIFuBWKeWPR3NMZwKFhLI+TBKYbCPIABYDGEPs6wV2EWAnQd5AP2OwOYBfOUI9U8HDlKsMXgMHQQ5DOaAHew945wlj4TIGG0c5PgdUzaMKbJaQYBo5Mng1jcZik1ahjcNj+gWSpcQ5QIAs0HuUbQD6MSh8zG1yPD0+RNo0yLYYrL23gpu/1sLCC+I4DnzotlakC9e9u5tvftTLkr77x3V0HPZxy7+1MPPsNJmUoHWbn0XLEzz4k2pWkELildZIYFLbkKN2XJ7f/XcdybjBTf/UQXebd6+yQ6KeBMmExS2rpvK5Ow8wvjHnrShyglvfMYlnHqoABGkMfEh8SFIqG7lWJejNJEMVLtXqPSuEtQokB5pDHLjDs807tuAXd4xl3KQcj/1PJZMWpfnkt1owTHANybY1YT773iaVSgYzrkxy2bv76Pq/IPntQ0VE6R2++Bc93POWacSSw53d3t+6Cy+Eg89gh6fxw7vvJhR2yGa8189e2Mw5i5rZvXsM9z54ETdc/zCf+Njb6VKzeoSgrHE8590QJRjczoSGLqLlKXbtnoxtm+zY1ai62EF4wgSEEPQ9/7xXvxswLMH0hSlM+2Fmzz5IPu/jyafOwZWC/QfGF/d9ucybs5t33fQnPv7pjx5xLCEklRVx+vqjx9j79GW0o4z+fjTPf6aymaBKnvJmnQNq9v9nKvAjmUeas8hyCD82nlMwh2AbQXJ4uQOFJusSyGGUtK3xyltIVQJi+MyxBpu0ClltwKaZAAaSg/jpw2KampFOUzbrniPqgHozbImgGpcmEtxF5RGN3wcZFMJbCUGrUP2Roe2gn73bglSPyTO+MY+dg50vhPnRbeNLxi7ZsznMHbdM5HPfP0AgKOnq8/H1DzdyFhkmYbNpWoCv/udePnL1WcT7DN69Ygat+z1h/Nhd1fR2mgyuDIb6Rg4dCpJNeT6JfA4sS7J9YwQDr9XmdDJEcfEj6cSiHIdF748RarRZ95nq4tW6eCu55STJAdsJklMFA6WEqXNT3Py1Fv7pmmmE6h2ECY4NZgDCMcnYpM2VX26naVaGSWdlkC6s/kEXqQ6TA/eG2PkzLy9EBAXB6SbBcJJD3QG++X97+eRbptGfHm7WAzvWzw/u28zPfjmBdX8p58Kr+/EHBlWKnYexY3q47OL1jBvTy5RJ+8nV5EkkQzgZGyPdy0c/fD9IsB2DUCjDxz58J4lkiM/c+iGSycEs5GxXF7guvooK8gMDlFfkuOOuXcUhSeHy2U/8FMcx+fy/fIDOrmpeDtHyBIFAjosu2EhlRYKF83fScrie/oFy8sqRvfTcLdzwxr/wsU999Cifx9MbbTI6A0gN+1DOJsOUomtz8DWvVpDDJGVL3kSIdZSp3gDe/zsIMYej2bcHf4aWWhtq7unARy8+OlTOq9erFw7hp19Vy9yLv9ijdzFpFpRcwSSynEuSMhy61HxmJhlex8Cwc5b+lAorWawQeuPHO7jl31tYsipOOmng80t+fvsYYt0WpaGtfd0+1j8S5S3zZ/OtTzUwdmKO+aRYrO7DnAMZdjwZYsXr+rnhw1207h/MO+ho8ZHPDf+aeeMxgQutOBOnZbnzjnq+8PYmBvpM5i5JqoSyHDFMQgGXqul5xpp5QobLWTckmfPmBOGQwwHhI4vXiOcN9DORHDsIFWsQBcNeofGW5gCNszJIS7LqTX0c2BngExdPZc33K4kssGmx/OzfHmTukiQVNQ7CgPJGm+rZOXq3DTbIHPdmuOGBfeQudfnqBydTPSWL7XcRQhZtGuEyh9mXVnL+qhai0RQrFj7Gji21Q9dvwms2VFZuM3vWPvK2ya233cNvNm/jbV+tpHrZMrIyyle+/i4cx8TvcwgEbNraa/n8lz4wRBkAWNEo1cuWUXfhhRihEAM9Pj567XRyaQPThIDfJp4I88WvvvdlKwOQfOHTP+b2277D7Fn7cBzBB997F7ff9h0uX7WO8eO6mNLUwiUrN1ARTbBsyRamNLUQCp5a09IrQchTkOwwUixevFhu2LDhuLf/qbHqJI7m1UEvJg8S5Y3D+t72YXI/0SOiQV7PgMoW8JKO1lCusoo9M81VxOjEYgMRhJqpexwttuRoETODMUQFMekwOI/3smThMmI4CHYSIIMolmIYg2dbX0h6SBHtFnw8UcglKI6rdFbuEa3K84Uf7ScQkjTOzGD5vH4JsXaLO79Xz4rX9fOpN59VLHZ35Mze+3suSaaSYy1lLCfBOiKMvzzjmXqEBCkwjNIGKkdzDMM19LM35Kc/7aMnaBKtsYm3G6Qdi6Aaw2dvOsTZn4zhr3D53q3jifeZfPz2Fp64v4J/u2UiKxrilO+XbCGk2op65wniUm/kmebm2ESIVECQzBbMeIOKcva5CWYvTnHtO7upG59Hul4+Bi7cf109Hc8HyYTLCSVjPD55Pk1Vu2nuMuhp83m9IB6pICsryPR6Nvw3vK+T93+xjXzW6+mcyxqEIi6uO9QObtsgVBMby3LJ503+dP8K7r1/RdGHU16e5Fu334HjGBiGpLungk/9880cC+m6tN13H2/5tOSNN22jvDw35PX9B8fyL197z5DPxIlQWRHnIx/6FePHdeP329i2yR/+tJIHHl7ON277L6qrY2QyPnw+G1ca+CyHn//vVax5cvHLOl8pP/ndWzGtl5dsJ4TYKKV8yUHoFcJrlKQqGdFMAIlgNwF6MYvFqL1sYYodeAsljUub3ZtQ3N4sEf4BXM5WpZGNonCBgpA5doesQSEk8MIlHaAMtxiyCJ4yyCHYrpLXOouVQwUd+NlHkPWEi6sI8Bq1NODZ4VesVqsFMdSJC5JsxiCVNJk2L40/KDFMsCxo3+nnwV/V8MzDFQBMOivDpW/qJRByGO6sFsDuGj9PEaYFP4fx4za4fO77B7jqxm58Pu9++IOSuoYs/oAzZByeDPRqG0kEVdKhv8zgbbe0852HduGvwBOkwsAVgj/uqOarH5vE3V+o5dE/VPHX+yv53lfGccctE3HygvX7y9hKgMP4cIoFRTz/Q8o1qcHmivIBfvjMTs65KEYw7KUHBkIujTPT3HRLG+/5fBtjJuS9cuEmZHoNDAvGXZAhGa0mHSojWVZBbyLFUy8E6W714bqCh39bTWLAJDR1NqGJExGmyV0/qOM/PzMRYXqrANMn2LohzP98sx4hoKvV+4wZBiSSIQxDks368Pkcnlk/t6gMACZPbKO7p5Lbbn8XP/r5NUTCGfy+wTi2sWO6OWfh9uL/wjAYe+WVPH/gEnr6vJWAEEXXAg8/svQYn83jo3+gnJbD9ViWQy5vYpoOW7dNxXVNbv3Ke9m9pwHTdDBNiSFcvv/j60ZEGZwqtEJ4jfI0ER6ggv2qNMFmQjxABZtU1E8TWa6hn6UkeSN9XEacK4kxtiRotJD7egEJrqGfevLEMZhMnnEqE3Y1A9RhFyuOTiZLFaVC1BOOAVwVZTM4O+5WJp/EkPUBPESUxyinpdiOcegq1kByGD/7isXcAqwjQjt+XBf+ep/nkK1v8CKU/H5XRTUJsmmTL9zURHebD9fxBEXLXj8f/4dp5DKCP/ywDpDs3xPi2Qei2LbAMEqVnuc3CZZL2n2eI3ujFSabNfnPTzYwZ3HK28eUZNMG1727m99u3cbC82IUlJo/4OLDZTUxojjc8vQB7tq1leve3YM/4DL73CSuI3ClwJGCF58tY93DUb7/0/Fk04J8XnDvz2qx8149ozQm+wipkoFDU/q8PAFw4wZr31vDNW/pIZc1CEYcchmDcLnD9271qocWCsMBfGP1DH7/j7PoPBRl//QFIGDL0kupXbKESQsjGKbaUAq+dd8eQuPG4quoQDree7/z+SCmKbHzAsty6Grxs29biH/76ATefu4svvyPC+nrK6cimuTFzdP40c+vJZUKMHfO3iHv9ZZt0/jk52/m4KGxPL1uPh++5RNDks5WX/FXbnrbfUOu27As4vEI3/nB9Uggm/UhhOC+B5fx9Pp5vOcdf2Tl+c/xcpk1cz8bn5/Jt77zFgZiEWbN3AdAIhkmmQphmq5qsSlpaa1/2ecZDbRCeI1yEXFV1XOQWaSLdu8wkhCSCeTxA5U4VOMMiTISwNUMMJF8sbF9lRL8FbisIk4AyTKSqom6oAoHH5Ia8iwnzusZoBbb+2KWVC91lSA7cukuhqmN4aYWWTSKbCfE00TYRIgDSjmAV9wMJJmUxT9+/RCNkzKUFrqoa8hRPyHPod0B2g/6aWjKUV5ll5xHIF1IJU2Q8IEvtXDjx9sBGDMpw6KVcdIJA9v1BHzeEaSyJpdc38/GNeVMmZXm3+9pZvbiJI//sZI//KCGTK+n9K7/QAf+oMuySwa49n86GLM0y4NvqqdrnyfkgmH49HcPMmVapjhey+fdg2DYJZcxcY/oJyCH5W0UVPlQVdq/wc+Td1cQLnN4/xdaMS3JtmcjrLjcCyGOd1ik+z2RMOX6AHvWVvPHgTfSVz+ejglTyIbLkJaPtv0mjm0gfN6Yv/KVGwBBtqsLX1UVtStXsmiVQ2dbkK998yb+cncDc5YkefqhSh7+XR2B6krsmvlEo2m+fseNfPu/b2DDc7P55Odv5pn1cxlOJJLi9tu+xVdu/W+qKmMEAjnecv2D3PTWP3PuOdsJBbO89513c9Nb/8y0KQcxVLXaRfN3IaXgL4+dS/9AGYvP3ollOSw5ZxurLn72iPMcL1/62nt49rlZjBnTw6c+fzOPPTG4Amic3MZDjyzjttvfRVd3pSqr8epBl654jWLhOYi9fE8vAqe6GFh44vRg0oWP/fiZW9IUZa2amRfyCjYTwkFQRZ4mtdq4nDh9GKyhXHVHOzJ3oJ688hP8LR+E91xWVfrMIDhIgLPI0IVVUoIBxjRkuCY8wH3fr2Xe+QladgeLznE7Z/D9L47jnp/WIgzJde/uxnWGdxSDS5f1cdO/tbPx05W011o0zUqz8poBrv9gFy/+NcIX39mkVg+wrD7G3CVJZixK4Q+4mBZ84649/Oi2cfzsdi9qqbre5j2fbyedNHnyzxVEp7SSbLVIHLLY8lCYUJPL4gvi+PySg/sC+AOSsoo8vZ3eSsR1C/4RjhhrGFdlfEu+eucevv3hCeR6TNX8R+V8+CTX3dzNmJocj99TiesKLEty7501HPhrkF3dIWorXWY2pXh8g0N11Tgy4XIAuiZOo1BX2vAHqV42G391NQObN5NJQKgCqs45B2GaCCF4ev+b+euXXKQI8MIDy7E/ucMbrevSsKiWyy5/Eb/fZmrTYXJ5Hx2dNf+fvbMOk6PK+v/nlrRPj7skk8nEiLsRYpAACcHddRe3XeRld3FZZIHF3SVAEjSBEAIJEHf3ScZd27vq/v6onp4ZAiz7Livv/vJ9nnlmprvrVnXJOfee8z3fg8/fVUq7HUMH7SQt1aq0HjZkG4uXDKcgr5o+vffHcxKjR2ympcVDYfcK1qztwycLDmfr9h7cce/FlJZlsXtvLlde9j7PPn4/4YhGdmY9Lzx5D9Goyu//cCUtLT9dh/NDtLa6mXn0t7jdARZ/MwJVNZg6aQVfLh7JjbdeTTRq3Yc3//GKXzzmfwoOJZX/i/FFrD5gEAGW4yaFKKPiDKFfhg042Bnr09ue/NWwWk0Owc9q3LgwqUYjioJKh1LpqT/og7wKJ7tw8sMQEFhS2t6Y9pD8wXs/nbCWFBNiOH7mkNQppyDQMJlGC5+SSCIG3QmxgR82S/mhY4oVs6VG8TWoeIXBsfZmogGFfeisxIPdYzJ782ae/mMun72eyqgjW1jxhZfe+QEeW7GLoF/gcMmOBKqE5QsTCPgV0rKiHDbSR8ivYJqSkrfcrLnL0v0xLo3w0etpvL1mC7pb8sStubQ0aHw3P+kH31+QQIRW9FhexwqhuWJqsEflNXHR4jKOLx6AA8mJ8SpiC4E8+LAsOe6UrdyxFRYzTcGkE+vZ8L2XhiodW2oqqaNHI9S/ncg02hqJ1FbiKOx30Ht1336LYrfj7dsXb92nFPf38dt7arDbI5imlUv46uuhrNvYh81bi+LbXfXbdxkycKd1dURHOEtKwdbt3RBC0K/PPoSASETh4cfP4vfXvUFdfSKPPXU60agaZxQJYTLzmKXMmP4dum6tJYNBnRdfO47Vaw8+5h/C6Qxy1mkLyMupQdejpKc3IU1BfYMXVTPISGvmf+64jIrKf16I6F+RVD60QvgvxuTYnFFgKXz+b1y/jQ4d/85UTBXJUjwEUckiEFdANYBAbAa/oJNDSsagEQ1bjA7ZzoTRkKQRjjuCg6PgcLDhJv5/PRoGVlLcHlPaLMOGD5VPSUTF0jvq7Ax0u+Sky2p55/F02qOmQljJXl016VMfpDA1SFm9g8qAjYVYs2QVCPgUZhYORAirZqDXYB/bvnCTURji3fvTqajUufDWKrwZpuUQBISCgjFHtWJ3migKuBJixVPpJjuwI7ySHW96CQUVXnoom+SUKDPPqye7MIh+o8HXH6bEjtvkME+A9HqDPdgZip8QChtwcs451diPjHDHhT34pJflZMKKsITmTDgmJpe3ucxOlmJSYVp3RnugSdVNzJDC4jkp8fMdrq+n8tNPUZxOso488mfvk7Tod+zYbmAv6H2QA0kdOxahKLhdAR57dRdNjc5Ywt8y9OGwhikFV142myuu+x2GaZmlN9+ZTnpqI7k5ddb1iSWHGxs9OOwRsrLqiEZVFMVA101+d+0bRCIq3gQfd//xGSJRlRtuvpY2nwspFT6dP55jjlpGOAyqalJRmf6LnAHA0EE7GDd6E6vW9GHYkB3xlWFWZgP+gB0p4Zwz5lNalsnuPXmsXHNw6Ov/Ag45hP9idL64f2+oqBaNCIKSTt0M2hUvJTLWghJAsi3WYMVaGXQ0bmlEw45V+LYKN90JUUSIeSTFtrSSzaPw8zUeWmKtN4m/2+EatC6OSeLG4HB87MWOBhxHMzVofImXo2imEY3VuH4gbWFh2BEtXHBzFSu/9NLWIPCmmVx9Ryl6ouTV+7O54OxKCiaF+OL0NKqWOS3eJiYAACAASURBVBmGn3XtY0nLMLs8UZrrbZTvcdCMyrdLkvAvsVFAmA/n5nDMZ9V4ehg4XCbrlybw/lMZPD5/NwDhoGDTcjefPZ/KAWzUtuhxqupHL6dhs0sGjPXRXKfTUNtxniMhlfUhN6PwMTq20rNhMh4f+2e72VmuYUQFxETG25k1TgzcmDSkZNLU4Ke2S/N569xEQu3XrTMl1UJCz54/eo9Iw8C36XvMcJTaxihCQPOKr1A0DUePfphRSaihgSlnwzFHfY/TGUJKSEm1jr19xq8oBlMmrgHgnDPnU1GZztZthZRVZDL/i7FcdN6HnVoBCDZsLmbyEWtpbXWwc3d+LE5v1ULY7ZYURn2Dl4cfP5O2TvUKPXuWousRFiwcQ3ZWHYMG7CIhoY3W1p8OF5156gIK8qrJzqrDNCE/r4ZwRMNui8Q7s7ldVq6ud/EBiosOsG9/58LG/1s4lFQ+hINgYrGUviYhLi0NxI11uMtt08F5L4glljsjHKM/NqKxHxv1serZwwjQiyA+VASS0bTFTFNXRk/7aNEujkLgQyWAYDh+9mFjPU7Wx8JR63HREG8P0+EMTvpNDTc+VsrFt1VhmnD9g6Xc9sABHnl/F8Ujg/ToF+KO10roNjWEokPvc32kHBZmUI6vk0qrJBJSaK63DPWiD5IBgQeDXoRwINlv2ljybSJXTS8m3CK49qFyBoz1oSiwb7sdzWb1TP52UyJ+BMX9ffFeMAJ4bvEOcnuEuGRiHxprdNJywmTmh0lIijJiUgvrcMUF6KJAKwoHQjZ2b3Fx/V/28/w3O61CsVhI6FiaCTtctKVmMt4WRXW7f+bqd71+zvx83IWF/FhoOS2tmSnTd9JW1UIkpBAJKwQb/CQn1NJ7YANte/bg37eP0tI0kpJayEhvQNM60QViBrX9NSnh8LHrOeWERSQmWrTm4UO3EY1qRKMqhmFVmR8xfh1Sgtsdol+fEnTdRFU76KWmKQgE7VRWdYhhZqQ30FDv5fZ7LuG9uVN55oUTMU2FAf32/My5gJL92RT1KMPj8aMo1jh2W4SWFjf7S7MRAkJhLX78S78fjN/v4Nor3vnZcf9TcWiFcAgHQQGOppklJFBDR9XuEbQSxOpZ3NVwWOyiYfjZGyOXdlQtWxXOCpIGNBaTiIbksFildHHMiG6LC6R1DRx1DR9Z77UXnpVjIzcmhbG1k8BaDXrsuK1QUnsjFyMimHJSY9z49hwUwIzCyj8kMfq+priBImZY8o4MUjgrwAu3ZsErgqkn1/P9gkT8bQc/NgUx8Y4gguWmm42POUjJjrLy6UR8WYIMGeH12zJ546VM+gz1cfiMZizHprFns8UR8iRGaWvWqK/WyO0RIhISlO7uOC/5PYNsXukhIgRfyQSrXzGwM9avYPqUeroVBykoDtFvhI8tK60Sw4DuoKzvUBy+VjaMnYax9DuSMyI01WqddHjkD35bCFZZ7Kq6pUtJ6NMHR0ZHjDwrq5FLr66guU5l8dxkK84PmFHBilcPEA2GQVFI8y/EaQuw8Xs3g8b54ongznkBsHIJUsKf7rmUikrLmM/5aBKfLhhLQ1MiWRn1XH7pBzjtAYSK1TZTsY7ZGkvw8WfjkRJmHvMdNj0Sp6j+5qI5lBzI5quvhzPx8DVkZdajKJJpRy7HZovS5nOyem1ffugQv18xCLsjwuknf4EqTaSERV+PYPIRq/nos8OJRBR69SynodFDSnIbQwbtJDurnuKiUjxuf5cVyj+Kf0W299AK4RAOgsQSoyOWtGxXuHdiUkgkJjnd2YBYgmsCqzey9VkrceeJ6f235wwEkr4EqIyFnLyxBLQbkyy6VpU640qfXcM+KjAQPyNiYZN+hBiFLy6FrSIZiY8ZNGPrRGL95LVUjIggEiNJhUOCheenEfUpCAXqNmhIk3iiE6D8GzsfvZJOvjfIgDE+CvsF4uejXao7N0bvXYGLuSQRRaGlWeP4i2oZcU0zf/1THoveSWH/IotFs32tm+fvzAUkHqKkE+XC06p4dcU2xkxr4s1HMtm71fqsqkt+91gJvYf4Kd3tIOBTkVLQLFTKXTaKCNEfP2k5Ec6/pYrigUEMAx6as4fPSjdy8W0VbBg8ntqcQppTMgm4PHhy0hh3USG9h4ZASFTVKuDzJFphH6EI0idPBiFw5udj+NuINDXhLylBShlfLWzZVsTdd5zEmm+8OD0mfYb5kKagzUgjGggyfGIzk46rY9Z5ldgc1vsV+6ymN1JCNKrw1HMn4fNZIcdA0IYQ4HF3EB/KKzJQVBjcay3fPLmbR2/MsfIOQQGm5Ilbcti5JZm2NgdSwqwZS1m1th9X3XgDqmowfOhWRo/YREF+NSOHbeXwces498zPmDppJYoiycmq47yzPmPShDX8VIhn8ICdqIrJnr25aJqkuKgMVZUcOWUFLqd17b1eP1KCN8FPr56lCAGPPfgwzz9xD2NGbfzRcf8TccghHMJBqENjAV5aUBlEgGNpJpEo9Wi0xmmjImZsrTh2t1ggKZMoo2ljDD7yCcVe7YhLS2ATLtbhZGUs0VuHxlpcXfISFr20syicBRVJFGJ9fjvQHAtFOTEwYiGlndjxoZGYEqXP0Damn9lghWscUL7Pht0paRwu+LLOy9alTt58M4snb7MMzs71Tu6/poC77yogqgqqgjae+VMuO9a5Oer0etKzw/QhSAZRgqh0I0wpNnSHyR+e38cDs/cw+cQmhICnPt3BxS+U05qv8MMVUBCFgfj5eEkqjbUa+T1CVJfauPX0HoDAiAi+m5/E7a/sY/D4tvh2ui4ZNqaFJEwUoLbCxpezk4lGBKpqGczVXyXw7pNZhFwJoCi0ZOQgVJXeR/egMHE5O9fbyesRonigVVl+4qW1pPfyIE1o3bkTpMQb3cbFp74IQhCsrqby44+pWrAAGUtQlG9q4dhz63jhm+08NGcPV91fSmZKFcJu5+zrq7n5yQN06xVECFj+eRJXHVNMJOb3DUPgdgdwu4O88OoMbrj5alas7hcPF7Xj2GnfcvLpK3FkpDHjrGpUDb6bbymJHnGSn48WTmPeJxOoq09CSjju2KUMH7KNEcO3cukFc7nkgnmYJuh6lCMnryIQsMWTwqYp+Pb7gTz02Fk/+Txs3NyT1986mgNlmRiGoLB7BYoiyUhrIi+3jgNl6UgpushyCAGGofL1kqGsXH3YT4799+BfkZE4RDs9hDgaUAmgsBcbpdjoGzN4CRh4O0XRK9GoQ6M/QZpQ2IGD0TEBumYUFuIlmwj7sdOhK9RhCAsJ0YhGKyqn0GgZtJiMdvtnHDG3Y9UtWCqpTagcRxM7ceBDZWysqQ7A+pigW09C7MROCMFAJUDTTEgpCnPOjTWEQwJNl5gGaLo1S/387WSeuTGXM/rXkHZ2mMHT2vC3Ktx1SXdaGjQaa9tDZta3VxRwOE3MIAw0AhQRJoRARzKHJCIIfntXObMurI8bCNMAX7XKdaf3jIWAIK8wSLhcIRRWUJAYKQKnLcox5zfw8v0HJyVnnFfLko+T8bUoJCQZNNXpDJ/UzNrF3niITgiJpkuGTWzljy+WcO1xvWjOOB7VZgMhMKNRFM0Kd9Uumo/H7ef1ldvQdMnSjxN5+k85NFTrXfar6hIjInC6DQJ+DcVmI2XkSGzJFpOpf79dXHTOXO66uIA/PLsXl9fgvBnH0rClHHuikxsf2kl6lp9QQOH1h7LYvMLNuKObOe6COjK7GSh2B+lpzWzcUkRNTQqbt/Zg05Zi3K4Al100B4c9TGH3CgACATt2PYhuk+za5KSmyolP7QkIpk1dQTSqoGlm/Pe+kiy27+zO9COXx69FVXUKn30+hgvO+RRTgiJg05Yi/vLEmT/7bNx0/atkZ9bzyBOnc/utL8aorirLV/Xn/XmTefSBvxA1VDTVWhmHw5ae0ZU33Egg4Pibz94vwYvvn4l2iHZ6CP8qbMRJRbzITLADB9sQ9CXAkJg+kQByiJITC+ekYNKNCCtxMYQA+7ATRiGHCE5MduGIVxC3z25LsMfWGPAeyQhkrGOaVY9g0tFm0jLHVv5BImhAZxAHq0cO7qSf1Lu9QtsUpAQivPmXTFoaNS79UwWKYhloGZM56j0kwAnX1jHkyFY8foPUzCh/vqqQ0t0OpEGMp98OgWlCMKCQYYZZg5vCTg1kxtDGErw8/YdcvMkGE49vAmnta85xmZRWONBsJtGwoK1JJRK2Ggul54Q57/pq3nk8kznPp6OoMt6nWQjJ5JMbGDO9mYYanfN+X0V6ToQnbspl3IxmKrY6qKqzYRqg2yTpuWGy8kOWptP0Rj5eFkK12zEjEao+/xzN7bZCPqEwrRGVK6cXU12m0604SFOdFovFdyT1FcVk8qlNXP1AObouefnxPizd3tEzeev2Hlx/6UwSo+tJSjP4+sNENF0lddhh6On52G1bePDqAsr3OmK6UPDd/ES+m5/EpbeXc9KldQAM6r+baFRl2/ZCfn/da8yeM4WGRi8Txq2PG3OPJxD/u3hggB6HBXjxpTSWrx1GY1MCx8/4Ju4Qtu8soH+/ElKSrYI2w7RCfJkZDZx75nxMUyClICoFffuUYLeHCYW69nD2etu49IJ52PQIhd0rME2F31w0FyEgENBxOiNkpjfQq+cB/H4Hz78yiysvm42iSD79fAyzjl3KgH57WLnm11kh/CtwyCEcQhwTaGM1LkpiYnICGIqvw8D+BLZjpwqdPTjiYnfLccdXBu3VtYMJ0IDKgbiknlVUVRwbX8XkSFqoR2MjLlSspjBgzaIn0kJ2PK/wy7D6Cy8JCSYHdtlRNSs/oMbueiGge58ghX2riIQ7WCp3vVaCacJL92Xz4YvppKaGaWjQkRJUTeJwmTS1WPUPO3CQrEfp1i2IfbeGCwM/gqETWgn6FVp2aWQNCZN5TIg+a3xM9LaQMTnE8y9l0dwYm61X2Jj/Riouj0Ftpd5JmkIipWDfZicfNuhMPbmBzJwIzgSTGx4tRbOBERI8cE03NN0kHBI01Gh4UwyEgJZACjbRgoyoBMqtfgHRtrZYJtdyySXbrdnr9nVuugYlOuiofYb4kSgsnJvBvKdcpE7u+NT0I5dxygmLMaKCaFRhzLRW3N7FvHBXLqXb9pF0VZRB43xUldowIu3TAGvs5+/MYdLxzaRkRDBNhfsfPg+Px0ff3vsZN3ojr7wxk9SUFg7ruxfTFJimgmGAw2EQDms8/uAEtuwfzPRp35OTXYfdHiFqCOz2KH177wfA67VWkYqQmBKiERVFkaiqxDQlkYjO86/MOsgZAPh8Tlpa3IweuTnmiEwyMxpobnFxx30X072gkkkT1rB+Yy+u39yTcNjGnx89l4rKNHw+F6vWHIbf/+usDv5VOOQQ/j9GCMF6nIyMhXvaq5CjsYfWgE6muyvCCBbgJYIgjIhlCmQ8fKEgsWPQjRD9CbImRpUcj48FqDSgxky95SjalTp9qBQTppgwn5AYG80aN6XTWkOokpyJQcpjiVqTH0+IDSTA3Yt20xpWqTqgk5Bk4kk0ujBdKvfr2BwmCUkmmm4Zi9adOoVJQU69vJqZ59Vx7uh+JKdH6DvMz/cL2quHYQNOpp/SwHG3VPPewBzCUqEnQba+5eGZl7LpURvm6PMaiG5QGb41SPZpEUac20zp4y5qsLGAREvme72bh+fu4oYTe6KqJoYp8BKlVWrs3eZk7zbB6sUJKKqk/3Af97xricAtnZ+MqiuYmhuXvZnB432cfnUNUsKFN+ziEnUH899M4bGb860v3CVELFE1Ge/U1hlCwMBxbezfbuel+7I59twGvlhzEglDOz6nKCafLxyDxx1kyqSV2HQD01TI7CZpbPZi+HxcfUzxj1wVgW6T3P7yXlIyIrS2OUnwBLj5hldRVYvJM3HCWsaPXY9Nt/oYNDR66V5Qha5DWXkaebl19BkWZFu5YNaxS/EH7Hz+5WgOlGZy8fkfxXME7d9FSrj3zxdQXpnBk488SCisoWtRtmwvZM26vj9yjFYO4LmXTyA9rZGeReWYJvj9Tq676XqkFDQ2JrJuQ58u2+zaXRD/u6o67UfH/U/GoaTyfxHa4/s/lhUygW/wdJGMLovN6ps61RpYXctCnEAT2USo/4k5g46kN0HCsUSxGefcEA/7jMTPEILowGj85BChEo0GVDKJ0oMQUQRzSWIZHkCyDA/zSGQFLtpQGEyASTERvbpOx5I9PsTU1+twZUWJAh+SiO9HbmcN2PSUl/tnduOi8X25aHzvThII1u992x188Ex6XLZaKLD8lmSaX3Zw9BGNJKYa/OGFEl7+fju/ubOCzHwrZNVvhJ/hk1s47fwaXMkmfc5t48rJlUxIa+Pr+1NpqdHIlxG2v+IhpV+Efpe00veyFoQGQ25qYdyVTfQc4qc96b53qxOkIKcwhDfZQOoCp91Ej3UZUzRJVkGYK+4vt0JREs64uorT73BjRqIkZ4Y5fGYzirCMYDSqsnZdT956uiiWiekKIaz2mlYDn85FaVYTIU01SUg28LWo3HZ2IbaG9egJVtW2wxHikfsfxePx8+2yQeiaQdCvomkm2Xk+Lrmrji5ESVUFAZpuottNBo5tY9jENkth1RQ0NHqIRNR4RbJAogjJmnV9uPmPV3DnfRexflMRr789nbffmwbA4WM2cOdtz6GqJoleH316lTBh/Dpefv1YwhEtfn0jURUhYOrkFRzWdy9CmHz19XD2l2ZzWN+9cTG8H4NNj9C9WyVNTR4qKtPxeAJkZdb95Of/r+PQCuG/CM2oLMbL0TTHVUnbUYdGOTbKCVOHFkvqWvP6JXhihWVhptAa32YSbfwUBFasvgKdSvS4E0jEYCABVuGmHJ2cTnLaa3FhQzKRtpizUUkiymZc8ZWFiaWE2pMQw/CjAaVpKkfUt6BJWOj1cMasWvof14o0YfBNzexa56L7FjuNtQLXgY7ja0d1k069Q8fdaNC9bxBFgdZmBafLpK1ZITHFYMLMZvw+BZfHkpwYemsTTTtsNGzWyR4fYuz0FqSEZe94qS51oCiS3z12gJzuYSJ+Sx11xB1NqDZYdHUKn7+fSACVGnQSMcifFiB/ajBOaS0+08d38xOJLIaUzAgN1TpP/zGHXoPa2L/ByYiRBmPHVjHnvWHsLq8FLH7/GddUU1BshdhMCfm9otT6GykcnMRTc3Za1cmxGbHNFmHokN1MOaGAtx/urIlkYfjkZtZ+48WIReGyCkJk5EbQ7ZL133pY840Xu9NEKCqrF3uBVlKGV+HOy2LIwB0ken0M7LcVWbsHf5vKX2/OZfqZ9RQUh3jjTis5nZEfprbcBhj8/vH9qJqkoVZjxnn11nUSFmUTOgrVABRFEokI3p1zZDwpu25DHxyhEhpX70VKsNsjeBNrUWLbdCuoYuFXI+lZVIZNjxIK6TgcEcrK0unerYpB/XezcnV/Xnz1OL5fMRAhJAMO29OpidHBUFWDiso0HvjLOYSCdqZOXkkk8uuYze4FFTS3eP6jei8fWiH8F6AWjT3Y2IwDkGzGwR5s1KCxEQcfkRjrJiZZHWvqUo9KOBZH9qPQhkr63xmfN7CKwLyxtpsgGIyffCLMpIn+BGhDYSUuluOiFZV6NA6gsxIXX5GAQDAixhZqry0egY9UDLTYPi76qIL000MxmQcbPc/wkTkyhFCgx0l+jrm/jj5DfSwt8/I2KSyKaQ9twMFi3Ey5q4FL/lBhJapNyeePJXPnwB5cOaWYqE8h2xUiMTXKuqWeuEHKHhem6JQ2DvuNNYutKbOMwJSTGmlXHv39jB4016kITaJoVmOZO8/uxp/fz6cKHYHkGzx8SBK3ntudr99IIhqxdiAlPHlrLnu3uGlpsMY2DYWdG9yEUOk7PcikGxpp8lh1AWAxfhbO7kjohsM6lZVJ7N3mIpAxhYcfPwOJRX/sTIGsKZEkp0fiuZN2rFqUhBFVKB4YIDk9SmOtjbvf2Me9b+3jxaU7SM0MEwqIOMUU4DcXz+WFp+7l4vM/AuDccxbisrVw/pjeLJ6bzE2nFPG7k4pwuk2uvK/USt5LSWpWiIknNHH4zGZmXVhPTWlHzD4aVdixMx+j0xxGSkGbz019fVL8NafDz54Vraxd7KClQcXhsO5g07TO5+atRbz93jT2H8jiocfO5PlXZgGWowhHNBRVcvVvZ3PumZ+RlNSKlAobN/9YSCt+FPQsKqMgv4aiwgoMU+XzL8dQV5/8M9v8clx64TyOn/HNrzLWr4VDK4T/ApSisz1WrQqCMmyUYqcXFi1yD45YXkDEysYkvQiyGwdmzAgfQStpP1hV/C0owEh8FBBGBXoQIiE219exwkoBoBo9thqxrNRe7DhiInc7sGOLJZ17EWQXDg5gw+00qC9S8GYZZHcPM+qcZpZvSiDXH+TamcXYHSYnXlrLaVfW8PA1eSyam4LDZZCaGOGEgQ0k2A2GTm+gzaXg8JiMmd5C/mFBtqz2kLFJ0j8SwtwNCyZnYgQUpn1QzbwX0+nRL0hO9zD7PnTw/Q2pHLe4iop6HX+bRnKGD6fb5K7X9xH0K3z/mZfENMMqZouAosPF/1PJ5q86dJkUJAPw48AkrTiMzS5pbVJJSDI469oqPnszLZbYFQhhcu87e5GmoM/QIFIKHnhtDampTcx7s4ijZpVQX2Ult/1+B253kPTUemYe18jni49g89YifD4nLmcAiUI4rLJ2XW/GHF1KZn6Itx7NJj3PoLbMSqxK4MhTGrju4TJqynUWvZ8cZxhldwuTlGHQUKPxu8cP8OT/5OFrUXnmj9nkFEvyurfRVKOw/nsv404UbF0bYuVCHVWTVO63kZUf5p2/ZtJUq2FzSGRU4YKxfbjl6f30HhwgpzAcj+1rmkl+Xg2aBpEIGIaO3R7h3fenApDoaebAgm95d3GYUCARzSY5d1RfNF3y27sqGDktiMcdoFfxfjQtSn5uDeeeuQCwqKGaZmC3tWscJfDAI+fS9Ddm5Sce91V8NSAlXHDOxzQ0etm6rZA5H03+2W1/DoneVoYO2YGuRUlPbyQxsY2S0mwMQ2HZigFEOjX/+XfgkEP4L8BQAjgx2YSLKFYlb3/89IvRM8fRxlckxGfgQ/GzP9ZhOS+mDlqB/nc7BAHkEolHirN+ZIXhRHIMzXxEYqwhjoV2WmlLrPgsgShDCVBMCB8KA2e2MeXReiJhQSgoKCgO8dQXu2hpVDl7eB+K+gc56vR6bA6JzWVVI4eDCsffUsfMi+rZ+ryH/CFBzAS48YQi/G0qFSU2FAHL2jxWNbPNx4kP1eFINWlxqGxa5uHreUmceW0NBUcHSSyqwZlhUJxjgOioXh4xuZVgi2DkWCuMZIQh3CRwpEuyeofp3jPIht1uiIXR8oiQrEbRW+D287uxdqmX3z1WQl21jZZGNSZCZzGKWupVjpjVEp/h5+Y1IgSccsEeNE2Skm6dY5criJTgcErCqsmjDz4GQiHBE2bTlh6Ulmdw9JErGDdmE+eN6ktGfoTf3llG1sAMKtZX8eztufQa7GPIhDYevi6Pr+clc/nd5ThcEPAraKrJgZ12Bh+XzJSTNrH2mwS+fD+F1maVDUsd9Chu4oNn81j4XhK3vRNky3IHDo+g7xAf65YmUDzIz5KPkzENQTRiJa2nnNRA995WkVrJ/gxqalMYNmQHqipxOEIx5wC6HqFkfxYVVVZStk+fUk6atIcHry7ANAVGVCB1k+GTWuk1IorDHsYwFGpqkjnnjE/Jz7WS6pGIis1mxDvBmdLiztXWpfzNe7u6NgWnI4LdFrHCWgk+nI4w8z6e+Hc9Iz+E1+vj1BMWYbOFkVJgs4U55/T51DcksmpNv3+7Q/i3FqYJIaYDj2HZsBeklPf/3OcPFab9NLbgYAPOOG9+AAEGxBzCGpzsxEERIfZiJ5MIeUTIIEIiJpVoGAjyOsX7fym+wYMTk5E/02chiGAOSdhi+QGD9nSjlcjMIMJYfDh/wGgqON5HwyBBZYmD39xZzoGdDm47pwcN1RpTTmpgySfJXHZ7Oc31Gns2Odm12UlWXpgTejSw660EVIfJyZsqeOeJDN75aybS7JDPyCDKeFoJTDNZujKJcFAQDlltD51uk8J+Qf48ew9tZQpLr05lypt16E6LkbNro4O8gjB2t4mid8S+jTBsX+/ij+d3R9OgqU5DBfrm+ika5+eT2e0u1zoOu9MgFOhwkqlZYaad1sCUkxvI7hZBKNa4kRDo9q5hoHYj1+6kjCgsX+iltUVjzNFBAs1R5r+bjcMW4LU/Z6HaVSaebhKJ2vHmuumWvIlpZ9TTUK1zwbj+GGGDC24up8eAMK+9fyqzTtzAuNHrCPgUktPCBP0K1aU6G753s2pRIu5Eg5VfeomEBSOmtFBXqXPhLZUMOdzH7CfTGHVkKw9eXcCezU4UFTyJBu9s2BIPZ0lpdR7T9QjXXfEOTmco/vqyFYexel0/Nm4u5ubrX6OoRxlSwoNX57N4bjI2u0k4rPDE4jKKejbEz0u7KTMMBdNU0PWOCcr2XQUsXzGAs077nOtvuQbfT2gMHTl5BaeeuAihmCDpwlZ64tmTWbv+xxlJfw+Sk1q4509P43RaJdv7SzO598HzCYcPpr52xr+iMO1ncwhCCK8QouhHXh/4vzqqrmOowJPA0UA/4AwhxC8TJz+Eg9CGQi+CnEwjvWMqou3IJcI0WhiJnxk0U0SIYkLxdpnZRP8uZ2ACjajUo1KJzn5s1KLyMV6CP0JSjSLoQYhcwiTG2ml25qPXoDOPZPbT9YHYtc7FJ6+l8+X7yUgJLY0q9VWWINuX76cQDir89eY8Xnswi5FTW3j2yx1cfm857QbXWxjF4ZaccEY9ww5vpb0mQoEYcwlsn6ukeCNWoZLZzkWFs6+rYsUfkpgzOpeoT8HullTvseFvVsgrCvHm7zKRpuUEzBA07dBQVHj7r5m0Nem01scqnFXJ5nI3aqLErpidzo7E5ThNMgAAIABJREFUk2QyYWYjCckRktMj3PrMfs67qRq701otVJQ4YjIaLqIREXcC7Q6ofK/VQzoaAX+bwp0Xdaf/SD/eBD/N9Spzn07g1QeyLBZY2GDhy7B0jsa4CXs5+sx6duzsTnJaBKcrzPPfbGfpZ8m88VAmu+au4tvFhYRDgsTkMIoCdqdJdrcw65Z62bTCzTcfJhPwqUQjCss/T6SmzIYnSbBuQy9mXNCMP5LEni1OivoHGDy+leZ6jZLtjrjR37i5kAOlmaSlNHVxBgBjRm3h4vM+ItHbxguvHkdlVSqRiMrKRV7yikIcc249UsL/XDeLbdu7d9xnUZVgyIaimNhsHc5ASijqXobbHeC6m679SWcAsOS7Iaxa2wcjqqLG+kZHoypSwqABu/7Gk/HLYJoChyNs9YEwBSnJLYTD/96VQTt+0iEIIU4FtgMfCCG2CCFGdHr7lV9h3yOB3VLKvVLKMPAOMOtXGPf/S4zCz3ACaMAwAozuJOuQRTTO4ffEZar/92hEZT5evsAbN+sL8dKKFpOg7kA1GtuxU0iYcmw0oNGdEJ0lrgGG4aNbTNyuDYXZJDO3NIXacp1QQGFmj4HcdnYPbBid+gdbUBRJVl2UeSNykFtVEnvFvp8Cm55IYO6ELNZ9k0ByUpRMEcFAxCmsLkyue6gMIyKwOw1MU3DK5TUMGO0j/0ir+llGBKse8PLF1AyuGN+LBW+mcsT0JlBg31wXQgNXtlXWfPtL+zj2nLr4zFLR4Kr7SrnwliqeXb4DKdoL4wTF45PZvdnJayu2886GrfTsH8CIQmpWFE2XvPdkKqEgHDbKj27rupKXJuQVWedr9yYnCUkmPQf4ee/JNBZ9kIyiwHNf70C3S6QpMAyBnpzM019up7Yhg9vvuYSHHz+bjz8exlX3l9PapLJ7k5PWRpWLbisnR1/KXRd1s6p5IxY99ek/5rL8i0RCAYXE1EhMYtui6T65cA89+/vQ9QhSahT1rOG6h0t57JNd3P3GPv7n2RJSMiwjLQS0tHi58eo348npdifXLif9P3f8lobGRKprUvnm26HousENfynlyS928pvbK7nn3XLsqcl061ZJOKLQ2uZE1w1KyzLi47XvSwjQdZNJE1ZjmD8/ww6FbLw/dwq6HiUSsRzBt98P4p4/X8i27YU/u+0vRXpaEzt2FXDrHb/lvofPo7HRy7gxG3C5Aj+7nRAHT7Z+bfzcCuFWYJiUcjBwAfC6EOKE9mP7FfadC7GWvxbKYq8dwn84UjE4klYkYCCI/iBh/C7JfIulub8x1oJzEQnxDEUJ9vhY7dULmURYg5M2FNyYDMOHMCESVuKzd9OEMCoeLG5lexK9txkkc1gYR6rB0itTWX27xQJp3GJj7b1JyLDFfjqqpYVJso2xtMWJrgj47IlUAI49pQ6Xy2DO02l880oi+z+xZpIlOx3c9lg3PvYkUl1v4/nbc3jipVwum9KLZa8nsuCkNHRPTII5arGNIoZAUSWRsCAhOYrukGxc5gYpcGblothsrPrUIL9nhJWLPDTVqeh2iapZxn7BO0lsXObmrGGHcdvZhSxb6OH5u7KJRmIsophU9Jpvk7hmRk/uvyKfxlobX76XzINXF/D9Ai+mIQgHFezO2HcNtVJQ0IASbqKiKpVLL5xDiraH+35TwO9O6onNLmmo0Xjx7hw+fDGNIeNb0DTJ0k+TUFQYd3QzdqeJK8HEiFrfLzU7gmkIXnrucAxT0L/fPp587kRsdsm00xrRdCusNWFmM0lpHbN2f8BOckpz3HG2rxCktO6k5k49jkcM3UokqjF8ig+b3aqRGDaunuee/SstLW4++3wsdluELxcPo7B7JXtLcmlt65iYtDsFjzvA4AE7SEluBkBVfjxnNmjgLgxDYc6HEymvyGDwoJ3sLcll+aoBv/gZ+Tns3pvPA4+cR319Erv35HPvQ+dz/lmfMmbkpl9l/H8EP5dUVqWUlQBSypVCiEnAJ0KIfP410twACCEuBS4FKCgo+BufPoR/NiwnwI82vGx3CgZWa8vPYoqpSnw7pdMo7X9Z4xxAZycObEgGEqSIMNtxxNg6FoyoiK8iehFkNW6cGBT2D5I9LkTh8X42PpoYH7vPha1Ur7DTuMVGb0Ic92UV+z91IR5ORLFJcicGqVpmx7lRMENrwvmGxjSzlXkk8eqfcphkWk17vsHDUaOaOPMv1Zw/1oohb1/nQlFh9NOlfHdpGh9Ny2Tqa3U4MwwO7LJz0mW1nHx5NS/enROnew49opUBJ/fBp3QjO72Sqt1Bek/3cMRxe7sYREWF6ac3sXOdm/lvpaBqkpKtTir36wQCOhCJtfwUFPSKgFD4+qNUpCGxOUyuuquUY85uYMN3HvqP9nPJnfWEW0M8/cccAj6FCZN3M2X6fTQ0evClGdz8VAkPXVNAOKgghCQhyaC1SaOmXKepQeWJW3P56NPx5CZs5uVlO0hOC/PAlfmcdmUtOd1CvPGXTPqPLgEJhgE3XvNW/Pv8MO8BFk104uHrsNmslVznz/j9OlHDRlJia5yf//o7R6MqJpdf+gHehFZsNsvBRSIqq9f2ZeyozaiagRCCpd8NjCmbRvAmBLrs226P8JuL57G/NJM77r2E++58imdePIG9+/K63OMbNhazZWsPampTWPjVKHoUlv/0A/EPoGePUvr320NaahNCSKZNXU6Cx4/P72ThVyP51+ibdsXPOYRWIUSRlHIPgJSyUggxEZgH/BpqTeVAfqf/82KvdYGU8jngObCSyr/Cfg/hH8BO7KzBYtD8MOzTWdG0o2q483uykwuxXr/xLwd44Y4c9jY5EMA2nJRjw4FJMxqpRKiPyWI7MCjDKng6gA1Nk0SE4IudSSzp6+Xlpdvoe3EbjTs0Fp6WwdBbmjkw38Gy36egJ5gk9Y6ip7bx1ZPJDJvWyoQnGtjzgQt3rsGbp+TwLQlxnlSlqfMxXjRN4ouqlAdt+H0Kms2MCdZZyqnXXtETxQYXHV2N7jIRCjwwe2/8fP3usbK4IUxON3jo8XfZur07qcnN7NjVjZffmEmy/QDTZljN5FXNMl5zn0/j09dTEQrc/OQBUDXuvns63sTXiUYFkbDA6ZIs+8wFUjL+6Eb2bnVSvtdOQ7XOJ6+mMGJyKxfcUs7t1x3F2ecs4dGPdmN3Ssr22Ljv8m6UlngZMaGey+8uxYgKNN3EMAQJSVFam1T6DPWTnGZwzQOlvP9JPxbPL2DbKjt3vlrCDY+UYnNY/aKHTRP0H9SMqrY3uZFdnEHnv9tLGn7MGQAs+noUcz+e1OW1A6XZgNUWMzWlOT6exxNk5jHfxf+fNGE1pinYX5pJt/ya+Pbt70sJLa1OHvnrGfQq3k96WhPjx2ygrDwD01SIRi1z2NDYMakwTJVde/45E1G7PcQx075H0ywplfS0ZmbNWMrqtX1Y+NWof8o+/xZ+ziH8FlCEEP2klFsBpJStMWbQ6b/CvlcBxUKIQixHcDrw8xq0h/BvRzEh/Chsx2EVQSFjs/yumEYLG3BR1ekWG4UPJyZfkwBIMrIi1FfrNDXptDsTA0kjKn2JcBgtODDxoWBD4sKgHBtNqOzGQSSqgJA4HCbXP1KKy2sSDSq8+komQ2/xoaiS7rMCFJ1cHjfK9kST6/fut/6XUDjLj9Bg+NlNbP3ITkVLh3pTKxrCkKiayZ6dTi4/sjcuj4FhWP0GQkEF3WYy7cx6BgzzoXs7wh9gGaBwWEVgxnMAUsKevbn0mb4fr9dHWUU6xUVllO21UdAzTDRqqTysXWKFTKQpuP/qIk69rAKnV+WBv5zFCVM/4Os5yWxf52LfVquGYcWXiYSD1lrs3SfSmbN9C+uWJnDHRd0xjX08cWsedZU6p11RS1pOBEWRhFqjTJjZyPrvEtBskt8/foAVXybw9bxkPty9KZbchsNntDD+2PfYV5LNdScNYNOKWo46NUTAJ3C4JM/f34PpM0PMOKehS+y+s0RI+zlR1a4GuvP7QlhG8qegqiamtMKHmmZiSgiHdHTNQNOsCvNX3pzBd8sG8deHHsTtDh3Uo8DjDvDYnx+16MIGjB+zgUkT1lJTm8xNf7jyb97/qmrpNXV0mvvfY8u2ntxx38XceduzFpssqrD0u8G8/vYx//DY/1v8ZA5BSrlBSrkLmC2EuElYcAKPAJf/ozuWUkaBK4HPgW3AbCnlln903EP450IBehFCYgnhtbsCZzzRaxn2NgQ1aDFlJasozo4kEYMTz6hj8PhWoqbgjUeyUBSJ22uQnN6R7I4gyCCKF5NsoqRi4AR6EmYwgdh+JUjoO9zH0AmtCAXeeiKDxR+n8O0yL20+lXAYQkFL8lrEErqtTYol1yBA6FY8/oj7G3lu43Y0RcZyExIPBkKCEVWIRgXd+wR48osdJCRZsWebwxJim3KayaeX98Tnc3QJ/QDYbAaaLrtoys04+rtYQ/gwZ5yykITECJl5EXZtdHDeqD5sWeUiryiIbpcMHt/CigUuFn7eF7crSEFeDXde2J3jLqhj+KRWIhHrERZCUjRUYfamzbywxFptDJvYynNf7+bRj3fhdJu8/VgWD16bT8l2B3e/sRdvmuCDZzJ47vYcotLB3sbhXPtQGe9u3ILd2eHc2n9aG6KEqqsZdkQrG5e7+cuN+fhbFQYN3MvUkxvjNNnOxr6l1YWiwLxPxseNHvx4OElKmDJxDaeeuBA11lcgweOjV8/9AMyeM5XnXrJ4J5GIgqpAwG9H08yYDhJMmbiK6VOX4fFYjiUUUrqEr9rbdEppOSdTCiqrUnnkr2ccdK+rioEQXQkMl1wwj+NnfP1Tj8ffjYx0q6Vrc4sbTTUpyKvm3xEqascvka4YhRXa+R5rVl8BjPs1di6l/ExK2UtKWSSlvOfXGPMQ/vmoQEfFYgalEEVHkoDJKHxMoJUCQmzHGas56OD+LyGBZXjwzBMMKPAT9CsdTdF9Cr7mjttxL3bm42UJ7ngYx48ggKAaHQlkECE9N8LmFR7efyad1iaVI2Y1AZKVi7ycOqA/5448jJIFlhZO0G9RN0u2O7no8N40VGsoMSMW9QtemZJHxFRIJYozFvYyEei6lUTN7W5p/dRX6fQb7mPqSQ0YUUF9eZBdw8bw4msz4sdvCcfRqa9Bx0/7/+01BK4kK/yU0yNE8aAgs5/M4MOX0hlyeCt3v17COxs3c8rZG/FtWMbWj3ejaZKsggj5RUGiYYFQrKK8gcOqSUw1ycwPY3dKdJskt7ufNd948Ldaq4fvPkvimmN78cXsZFobTHbvTENz2hhxeC39eu7ENDXKa3IOmsULAYOG1VJ0mJ/fn1LETacU8c2HyZw/rh+tLQKbvesq0TShqclFXX0ir709narqNPaXZvLHuy5j9geTDxq7/UfTTMaO3sjJxy8iN6eGIycv5/JL3wMku3YXcMy07xFCEo5YFdvt3dXaDXdhtypOO3lRrDBNw243uwq8dro+waC1utiwqSfVNakHfea8sz7lpFlfARKP24/X28aQgTsZP3YDHrf/b7KCfgl0PcLCr0Zw461X89DjZ/3bdY1+SaVyBAgATsAB7JNS/rQ84CH81yOXMHmEcSLpQZhmVJI6VTnnEcWAWG8FeywJDRmEGYuP+oDGp2+lcfpVVcx+KjPeNCYc7wMg4rUOiUTjFRUrcaMgGYmfo2khGYO9CSp7hYM3Hs5i7vMZ9B1mNdqRSNIzQ9z5Sgn5vYIoKnz0YhpZ3cOMnd5MoE0hOVb1axqg2CSRMsEI2uhJmCjwPR5q0bjlqf188mQqqxYl0tasctOT+0Fz8ci1mWSO6M2SLwXhpERqaiUHSjPIy62JV8k+e3sWZ19fjcsjETEH8MMwRkJCGNMAl0fypxdKCAYEr/45i+ln1KPqkuQ0A4/XoK5aZ/saF7pdcsPxRZTssJNfHOSWp/Yz+8lMNq9ws2uTnW69wmiatb8Vi7y8/lB2p6snsTkl2d2jnH9TJeNnhcnMbibg06iszuDmP12B0xHinj8907FFe8FXFHoN8hMMqDTVWa1JX/p2O8qPMDkVBd549xhKy7KorbNYXzY9yoXnftwlD/DDlQJYfYmnTlqFaSpMmbgKXY8ya8YS2tpcVFalUZBXg9sViu8nGlVixyiJRFQUxUBVrfBOQ6OlT5UUcxydw1TBkI3auiQGD9zFux8cFd+/wxFCVUxGDNtGKKSzY1c3rr/qHUxTEA7ruJxBHn/oYaQU3PSHKw7SNnK7/T9b69AZK1f3Z+Xq/gBs3daDrdt6/KLt/ln4JQ5hFfAhMAJIA54RQpwkpTzln3pkh/AfCQl8iZeR+HDGFJKSfkTyYhd2UjDYgyVaF8VSQm1CjRWgSea/nYqmmIyb0szXC5JjEaCOJHQuYUbhZwd29mCnFRUB1MVk+RIwWLs9Id7HwdfSrspphZISmiQbrkrEvEoQrFd47b4sogh6D/GjqAIjKKhap9O6V6fbsQG69Q/hXBPbvaYzNtpGZEgqo6e14PC7uXpMOR6vwRHHNSNlC2Om1nDb2UH2rNUYdOoe1m3sS1JiGytXH8aylQO4+Lx5RMIqvlYNV0Kky4wYfhBbjz2JTU1OPO4AF9xSSWOtbiXhpdXy8953DrBrvc5Np/Rg80orx3DDo2UU9g1x0xMHiIStlYaiwLpvPRT0Eww53I+qSYxobKWmwIPv76HPkA6FUYCExCgJiRXcfuvzaJol92CaHTF/63/B1X8uJxRQuGZGTyr327n3t9244dEynK6u90A0ahm4YKiDYnygNIuMtEY8Hn9XmukPHINpKtTUJnLMtGXx4zvumCUYhsJDj51NaWkWs2YswWaLYhgKn8wfx6wZSzEMgaa1t7DU0DSDxkYvu/fkcdTUlQclutdv7MVbs48iNaUFgCPGryEzo4Gjj1qOaUIorGO3h7nuynes72QoOBwWy83vt/PEs6cc5Awy0hu4+0/PcP1N19L2C53CfxJ+iUO4SErZrhdRCcwSQpzzTzymQ/gPhAlswkEiBm2o7I4ZfAWJBuzATgZRkjEII1iHiySiJGAyEh/rcbIHO9twximrrXUaJoKS+S5O6V/H7M3tDUUsK1AXE8VbiwsQKJhIBEvwoALTaGYgAXII8xFJqP+PvfMOj6pM2/jvlKnpvYckkEIaoXcBpYpSxN5719V13W9dd9V117b7ra6ra+8VXUQBCyBVeu8lJJCEkJDeM/WU748zmSSAgq4Kfpv7unJlZjLnPe85M3me933KfaP5Q1QxeKk1y/T/TQsrb43EiyGwMxAH1m0a7Zh4Lj2ZCDQEs872Z4JpqbdhwosqSnh7hzBkUjEBUzVEGY4GpBDihjC9Bkcz/P3eZNpaTOzZaOXXT5cx5JxClrzcyO/uvxSnYLTT3H7F+TiK9nDbo53Fc8fSLHS81vE8JMTpDyc5WkT0BEP3V9fBavXy5Xux4C/khX4jDcPucRsKd5tXBLJyXhgrPgslMDWRseP2ovk7sA2a7s3LA0nPc3RSTXdZ4QcHGeNVVkUQHNBCYJDXH84xW3RaGiXumJhO7VEzH+3cw3vvjMHtrcWqO7pdhyTBv575K5u2ZPPS67MAKCuP46//uIq/PPSSj1eoe76lw2BLkkZ8XGO3+2P8aDS3BNI3sxSTSfET42WkH2b7zgzefPc8brz2M3L6lvDqW9MZNGAfQwftxWLxUlsXitnsJSS4HV03Koneft8I8XUI2Zw7aS2iqPPiazO5/qr52KxGTsvpMvPqm9O55frP8HpBEHXa2m3sK+xsVAsMcBAY6OCcMZswySqjRmxn+84MWloDcDi6N2ueyTipQ+jiDLq+9u5PM50enKnYjZU92Omgf6jAzBws2NCYRlM3B+D0pabakYjFyxFMjKcVNwK7sfkb0wSgL0764aR2twULOlF4CEFjP1Y0YBEhvlJVHc03roTGVFrYgJ1QH012Ni4ycCFIOkcTZPrFtzP4ySZiE73Ej3GiawLj1+novpXyqC5aDzm3tJA63ckblw8koqaCxsg4ZjxaSJ/Rrei6oQ8xYUKHIhh4ZROBoV5yh7Zx4S1HKRjVhizD3Q/vQg9u4Ju1MkcqY9DNQfzxrRpcbjM2m+e4UBF0Xxl3GFRVBcUjEBJhlCN6PYaBXbo0n9VLAhBMHoLSU2ndX8TDt+Rw432FpGa5efj6FA7ssNHWbJTpNh2oZF5JFNaEWAJ69UIKCICafSSkldLWIhESrp4wvu7xSERHNmEyqd0MtqqAPVAlItZLTYWFfVsDuPWuZUgSqKrgp3roep0b13Yv2RxYsA9BgObmAI4e1Ni3xc4FN9X7K7Iam4JwOK2kpVQdN5YgQEJ8DV5FYtPWvgwesI8jFZFUHo1k4+ZcWtsC+NcrFzF44B5cLhONjcHoOtTWhbJpaxY3X7eA0rIYdF0kNeUoUZGNSJLKr27/CJOsEBTcDrrAxTOX+q9dVQUcDivNLYFIssq/555NaEg7kyesIzi4jRZfA93N139KXs4hf1ntzPNXcsmspazdkMurb87kl4IettMefCcqMbEfC+4uq1LdVx5qQyUAjaUEoQMNyDT4jzSkNQ9jIQMXEmBHJwDNlyQ2JDqDfGY+BoVZNPEVwbgQuYRGqpFZRSCar0qp6/nNaFRjog6Z/jjJx0jwBaR6qc8SiJ3pIi7Rgw6Mf78O1S2wYGIMLQc7OGN0wvIUCJBIv6ydoGSVhHFO7OYAJt+9DytONK3TIJlklZZWGxazh3lfjuOOJ5Zz9JBIWnZnaeM5M2twOFpYtmoYAOaoOGJTVAICPP67cmzYCKCp0UJIqLvLChkkm47Jx8ezc20gucNd5OWXEzbwCiSbjQnDvqKmsJG6o2bS811Ulpq59dEKnvtdErvWG9coSBIx48cjmjp5cgKyMthX1srTA+u488lKzp7ZgMncfcdisah+jQHjR0AUdT57LYLpN9Rz9z/b2PxVC4FBKqIItbVB3HfbBTz+1IckpHVe63O/S+DL2XXETFAYc9YuLpm1BJPJg6IIhAS38/yLvdi8Iohp19UjScZ5Fy8dzvJvBjFy2DZuuObzbo5z0ZKhJMTX0r9fEV7F6NiOjm4kMaGO8LBWig4mYzF7ueFq4ziPR0YQIKdvCf37FVFSFsufn7yB5MQq/vC7N1BVgYaGMHbvSePssVv8XdNhYZ1ss6WH4+mdWoHZ7OGFV2bhdptYvLQ3pYdjyc48xPpN+ZhMXr5YOBKz2UtGn3LffHU2b8nkzXfP//Z/rjMQPQ6hB9+JQFRakHxi912bzCAShXrkE/6t47EZnUFdmFBrkEnAywAcbCSAamTi8LIXKzqG6hvARuyI6CTh4SAW/7h9cFGMjTmEYzgVgdm+x71x0VwcRH2xTHO9zIgpRmzYWSvy1YxoWkoMw+gxWwmwOJg8pxZzoIriEPAqEhf/bSeSrLNzVxp5uYeOS3ba7U4WLh6O02mlvDyKfevdpGW7u63yX35juj8Eoesiv3/kDh5/5F/ERDd22wkYfzd+m2SFtQtDGDml2f9aB5ndQ1enMOnyZu5/8Cbycg5iDjUEY5auHErduo2ERroRJXj2t0nsWh+AbIbfvVDGygVRrPvKjuZ2d3MI7e12vv4oBEVw0uBKRZIb/IZ/975U8rJL6GBR7ZhLh6GcdWu9sUpPqCX1Ng1dN4xyZGQrF11fTHSS4cCWfxrCkHNa6T/WxeaqsYiyzOatfenf7wCpiSX8/d4EHK0S+7fZEQS4d1o6VpvKpffUsfybgQAcPhILdDayiSJ8vWwIDY2hNDUFct1VXxh/V0WWrO7PytUDGDpoN73TjPBcdXUY4b7cgKYJLF0+iC8XjyAhvpYxo7chSzqDBuxn7/5UPvtiDHk5B4mMbEIUjRyMxythNqn0SqpCEKB3agX9couxWj089JfenH3WZqIim1m/KZ/c7IPcfdu//aE5XTfKjfPzDmKSFX/D2y8Bv5yZ9uC0IBiNKbTwCZ3C8jF4wUcQZ/5WCgvjuQeBfxOKBEymmTG0GSpXwNk+PiQPAlXHiOgUYyEEFS8CgWik4mY3NqzomNDw+ncN0OF4DmKUl4rA7k2BTOudhy1A5eXFhbSV+VTJBIHGyDhMlSX8+XeXc89v5xKe1I4oqihekd/cNpXirTB75WEsFqWbU5BESI/bTv++EJ3gIaNP90SqIHCcHKOuC0RFGtw5iiogS3qXJK1R4dTaKJLez+E3wPXVMuVFVvqPbgPRxMoFEQjhsHZDP/+4CZkSl1+j0S/b6IqedUsNGcPi2LQsjDHTmolMD2X77iQ07/FEhkEZGQRnZ5ORP5eDJQm8/dY4rrl2BTFRjeg6tDssBPqauo4tD9V1MJs0nxRoZ4jovClr0FR45r54Fs2O5KLba1D0ACSbET93uqy8+e55PPPU04RFGdTZHZ9f0U4bKZkuDlYYtCDXXTWfNevyEQRYstzg1Jxw9iZCgtsIDW0jtVel794adBQBdieXXriYnL6lKIroc1JNSL57bbF4aWm1c/P1n5KZXo7bbTjIWdOXc9lFX7N6XR4xMY0UH0xA0yGjT0UX5lud/YXJxEQ3kppSiShq/P2JZwgLbUNRBe7/1bt4FZm3P5jCVZd+5fsOgKpK7NzTG6fLetz9P5PR4xB6cFLUYlA5h6LQgkQrEtNpphWRNkRW0FE7bTiCQBTafMI3YHAb5ePw6x20I/IVwUynCRNg9YnoLCAEh2+HEOhzRDXIRPlKT5PxIAKZuPiE7tUdHgS/G9IQQDFOfFafFqxmHVO6CXehQnNYNM7AYOpjkqjwJuMWrYADRRGQZI3itS0461vQNbWbIewwhvPfiGTbgYEkD4nniT+90M1gahrce+dsVFXi47nnsHTFEARBo7Iqgo/nTCArs4RzJ61j6/Z0Vq3tz03XfMqbj0Vx+T3VBIerfmPS1iwTFGcB2ogYnMfBA9F4WrtXrMw8fwWDBhT6nw8a18rwSa3cdL8xn6yMMmavPILLaeKBR+6mzY8IAAAgAElEQVTyV/tERjTS7rDhdMq8+tYMPB4Tui7w5N+TeeapZ1AUiQC7+7hdTMd1aprB35+SXH3C70pdlQVBgPlvx+BxQsLQXaiRuQiCQH5eMYIgkDkxieqKWrZ/Y0MQwWTS+eeXRSiqiVFjioiPa6DscCx/fPRmjlTGAPDN6v70zTzE5ZcsMcgOdfB4TVgtXgYO2I+ATtHBBHolG7kHUdRpd5h5+C+3MGv6cvr0PsIzz1/Ozdd9SnbfEv+1fbloOJ8vHMne/Wms32iQ191/z7tkZZShagKypBEe3kJmxmH/PQkLbUMQwCTrZGWWcvhIDBWVEUY/iyIiijpHqyJ44+1pJ/6HOoPRo6ncg5PCgs4g2plCC+fTTIqPYC4YDSs6ob6sQAdz6WAcmNB9Cm06Yahk4kbDMNylmPEgchgzHgRqkFhHgC/0ZBzThshmbER26UMIQSMIjRofn1HH+B1d0Hq3XINOpKARsEvg5dH9KFeT0IGDuUPQENg7aAySrBIV08y7745jRlZ/ZqTn4apvRhAEZvXtxwVZuaxfHGwkeRX44Jlols0No353GdqBZT4D3hkm6Gg027Yjg9XrCnyvizz051vZW5jKOWM3s31XH55/+VJ27Mrknt/9hq8XZvPwtUbtueI1DG9KpoukpGY0De67Zy4vvfQiV136FbJs3Off3/8mBflF3RK+sglaGo31nSiCJOkIaLz8aCJnj9nMkIG7Abj95jl+HV+32yj/NZu8ZKYfJjjIyZbtWSeklIDO/IbF7OXhx27s9h3RNfjVeX3YsiIIXRdwt+uGXkRTDR20zQeKkvnTEzfQ1Ghn/xYrgSEqvXOdeNwiRTtt2Gwe4mKNENas6Su48dp5/Pqu9wE4UhmD2ycgIwhGWEeWFTQN6uuC0RFIS6nEJHc6cpOsMn7cRqprwnj2hUtxu82UV8RgNhn6FyaTyqHSBJxOG+s25KPrArpulK5+tXg4v/7dPezdn8rGzdlUVR/fuNbxmZtNKmNHbwfguZcu4pN5Y4mNacBzmtXPfgh6dgg9OCmiUIjyGf0ANAro7NAMR0VFoC8u+uBmNYGUY8aLSCpuWhCpR8aDwCoCqcbkl/LcSAAbEIjH49cn6Nq234bECXqesKERj4dhtONFYL3Pmbi7hK3CUKnVZTRAmxHEZVft5KPL+tIeHIbHbEEXJVRN4PZ7fouqSQRkltO0Y4cvaK2hagIFk1QGjm2lrcWM2aQwYkoL7/xvHKARFOSmusJMVFz36iFR1Jn9yQSfsYXM9DIuvmAJZovBH5SXfYiHH3gVp8vCcy9ciLuugSFXO5Bkgx9JEHVU1dAB7iCKW7chm7FnbaX0cBwXzviagABPNyPdsXL/+70J/PG1MlSvgMmis3OdnZWfWMkZsIMrL27G61JITqwmIryFPfvSUFWRvpklTJ28zpCmVGHooD1+x3ZsJVSHoQ0NcZCTdajba6IEM+5W+d+bDedgMutEZkVBr+H+e1NbZ+Q/PG6JiZcVcemdlYRGeJn/ZgT2wM5eV4PTyEN8fB295Gp+f//rpPaqQpS0bnORRN2fWJakzrl2jGE2q0yesIFde9L8obzsrBJ27+3N3HnjuPHaeWRnHWLLtu4qaI//7Tr/4/999kqsFjdTJq7H4bRgkhV/r0MH6hsDyc0+iKYJTBq/geaWQF549QJU9Yepm51O9OwQevAf43ya6Y+TIF+YJx8n59BCL9zUIzMQBzI6I2kjCq/fIUgYXc8jaWMazd3UxBJxM86XbzgWkaiMpQ2LjzJjDK04EMnESUpYEDLQxy7TNzOa1vQIFr/dRH2dQuuMvgSHtKME2P3WrkMwxZ6UhDncp7Xr47MQkwfxp5uzuDg/l+tHZ/HZ65H+OWzaksHjz97h91+6DodK4tA0kQKfspYkqkRENKLpAnExdZjNCrKs0Su5isPlsTjdVsZdAZOvaERRBSw2ncY6E2aziih2MoYOH7oXgOnnfYPVqhxnsDsw9ep6RBE+fyeCoh028oe1E5/qJjDQQ2Cwwl13LkAQdKwWD/fcMZvbbpzL8m8GsXptPl5F6taE1hVdz+P1igQEuBgyaB8Au/ak8vzLs3A6zVhko7w3JELB6xFxNDjpKuoy7dxVzJy2gqqaaKrkSYRGGDmaGTfUk9inM9dhrLx1VF8HsqLKiD61ua5OqPO9nfPeubuz01fTBDZszubp5y6ng4zu6X9ezjPPX0ZZeRwPP3YzH83p7FD+NpgtXtZuyGPT5ixq64L95wVQFIG87FKCg5xIkk5WRin5OcW0tgZ+x4hnLnocQg9+dIgYYabtvoayJiS/rkEyHhQEJF/ZaSpuTBjVRToCAT7VhDq+fbutA2KA5n+sWAOYSRPpwXZCk5MYG26hMS+H6b/aRMCEajQN5rwUxfhBn3Db5S8zctjO48dUVTz19ZjCwrD36gWaxo4lLrauCgZVpaHaxMIPOsMGSmsrKclHEQWDllnTDOH7Pzx6Czt39wEgJ/sQN127gBdfnYWqSqiqEfvesSud2XMmAiK3P7iPqGgHsq+GPyK60zAe280bFtqKyaR1C+l0PhYYcFYbn70WyWt/jue+mb159KZe3P/sYZ64I439W43kriiC2axQXRPOw4/dxK03zmXNhnxMsorLZTIqbDxSl5LT7nMwmXycQSlHAcjLKeGuWz+hti6M8mI7vYcFkTBlLL9/pYLg0K4raZ1xYzYzdvRWQKdffhGiqFNYlIzXK6Fq4nE9GWaL0RTXJ63C7wQ7rtfpNOPxmPwhLZfbhMcjk9O3xOCRUo3wY1ZGKV13nUYuxXiuaeIphXVaWgJ5451p7N7Xm5joRv/8On5v2JSNooj+5w8/dhOHSn98rS/9RE0jPzJ6QkY9+FGhAAsI6da3cBALB7ESjILNlwcowMFm7FRg8kt69vX1E7Qjshubv34J8D/Wgeghbsa9Xs/HBXHoqkBdXDJJJfvYkz8MW1sz+4aNp7m4mMdv64VsMtTVdqwJZNOyYOKzzJDR79hpgygSPmQIluhoBEHAnpgIooi7thbN5er21oC0NMxhYSiKm7nzx7BhUy5P/flfVB6Noqo6koH99zGw/34GD9yLrsM1V36O2axQfiSa4KB2sjJKkUQVVZNoaAwmIMDZbRXucBiNbCdaqX87B5DRGDbvjUisARq9cx1sXhZCe7MEgs7s56J55M0yf2+F2ewlKbGKPmkVjB+7EZfLzLsfTuGKS77yM4V2oOu5VFVEljvDOx6vRGNjMP988WLqG0IhGvLS9zDmvDq2Hz6b2roa/vg/byBJKqKo4fXKvPr84+i6wOvvnM+adQWEhbbQP38/V162CK8iIok6ZYejSYivRTRpqIpIc3MAEeEtVB6NJD6uDknWkCUVt1vGZFJZunwQ505aT4espyQZczQqirp+k74/LBYPbreZzVtz+Ci0lQtnLEWSdDRN4NW3ZnD+lNUIok5FZSQJ8XWkplQa9+IXiB6H0IPjUIyZQDRiUU7+5mMgA1m42HGMtrIZjbG0AgI2NCQgjmYfK1H3PEUwGiO6aEJ32EU5VMUcoJNxRRu2CJWkSU4OF0fhCA+jsSWW1rAoHPYgdEkiOCsLyWqlfd8OAERJZ/Ll9ZQwi9a2442DIAhYY2I65xtmVDFFDB9O1ZdfIoi6vxTRFByMLSEBj7ec1tYApk/9Bl03kphjR29hQME+8nJKUFWjTj+37yF0HZZ/05/1G/M5d9IaHv3jS1gsCiHBnR3THQ7AZjOS9k6npZsAfdcEr6KK7NmbRn7uQURRR5aNncOsW2opGNXMvDei2bUuiP3bDNlOt0OiutzEG//oy7QbW8juW8ptN85F1yEv5yBut5m83GL+/dl4rrvyi+McT1daCY9HxmRSUBQBk6wSHdVEu8PGNVd8Tn5OMXa7C12Hm6//DIfDQklZLH16G6t8i8XQKl62ciDrNxikbo1NwXgVM4IA6zfm0i+vmOioJmRZQ1FELBYve/enMHfeONZtzKdPWjm33viJUc30xVlcdMESAgLcfPDviVx5ySLAcAQLvx7GZ5+PPfUv7wkgCBpPPfo8L7w6iwPFvcjOKkGSdHbsSmdAwQGSEqvZf6AXr741nSMV0UyesA6P55eXTO5Aj0PoQTfowE7sBKMSS+sPGiMHFxW+LuIOc56Oi0B/VZABGXx6Cd8+l65ru3M/ryMszYPXaegrj3mhAclcz9KPJd5fbJT4qdZORySaZBTF4J5xOSREUSAnq8xfXvhdMJm8eL0mPHV1AAwc66Boh5WWZgsmX64hK6OUWdNXoKpG7XvfrBLycg+yYlU/Fnw5iglnb0CSvCiKzEuvz2TbjiwA5s4/G4/XxPSp3/jpHnQd6huCiAhv9Rt9i8XjC390xsk7O6c1KiqjiI5qIC62sz982nUGDcSdj1eSO8TB0/cl4XEJ7NtqZ/FH4fSbaOeDj0fwq9tnExba6ncyJfvtzHkvj7q2RGaeu5iQsBMrmoERctJ18HpNGLoOXi69cDF19SGYTAqSpPnKMhVCQ7wEBTqRRM1/nbKsMuHsjRQVJ7Nlu5HQLS2L5a/PXMm+wlTsdif33vEhJWXxlB6O5aZrF6AjsG5jPmBoEv/zxUuoronA7TYTEd7IgIJChgwy5FQ0zeg9GD501w92CLKsEBToIKXXUUJC2hkzahu1dWHs3Z/KJ/PO5khFDH0zS7DZ3GzdnuU/7qvFP4oywGlDj0PoAWBQTW/BjopRGlqHzNcEIQHDafMFcE4NOgaNhYDRzVyLiXLM9MN1skOPa3Nz2gOxO9pw2oN4/b50rnloFSHZGqKo4kFk4VdDmLdoOLosHWe9PE3NBMaFE5A7mLbiYpYvCiXWnXnSOWRnHeLaK77g2RcvZubV86g9InH2jFqcbQJffBDDmFnv8NyLl/D5V6NpagriyksXIkkaiiKhqiLbd2YyqP9+rFYvTqcR/klKqPY7BIAvFo4iN7uY9N4VqCqAQFu7Fbvdic3qo+XWBQ4Vx/Pu7Mn85u4PsNtdyLJOdU0IlUejyMwoIz6us9O4rlImOtFHDKJDa7OExyVgsWm4HCJF+yORkoM5WhWFEWLScLtlJEnluQdTOFpegRTQxLaVAYyd0eQfp+vORNehvCIKt9tMeu8KvzDN6BE7EEWd1WvzGTZkNx6P5GNNFWhtsxAR3rnb7Bhr4vh1SLJKa2tAN6I4h8PGvgMppCRXMmLoTnTd2GX9+q732bErnaUrhvhlNQG278pkzOjt2Kxu/xy9XpG6uh8etjlvyiqmT12NohiaHQP772PEsF1U14Tx9TKDmqTrnP+/oCep3AMA7GhoQD0GA6mGQC0yFjRfN/L3QwEOLqCRCbQymlaGdwkBnQxdxTZqEtLQgcJ+I6igFxV7Q5FNqt/gbN+VidcUcMKlbEhODsGDRiNZrYTk5hI2/Cx/OeiJYLW4CQxoZ8yorURFNRES3I5TiGHihVWYzQoh4V4uvb2CvfvS/MbGYvFgNnvxeiUsFi8Wi8K4s7YSHNzG3PljuPv++zhUatA1d8S3Z56/nOuvnkfv1EoEAWrrw5BlnV5JtX5nAMYuoHdaBbffNJfgYCeqKqFpEBHeQv9+xcTF1nehlwDZYpR8djxP7uPivGvq+ffu3dzwh6NY5HZ6JVVx3VXziYlupK1Z5KnfFvDxv6I4steLt7kZV2Ulm5cH4WwXWbUowj9e1wT2N6sHMHf+WOrrg/zKZoKgU1oWS1ZmCaKo4faY/Y4kPKy9W2K643d67wpuu/FTLpqx9LjPoro6nL5ZJX4ZTLNZIaPPYerqQ+CY7+OhkkQqj0b6dJFFNE3i7Q/O42/P/nBS5vlfjOGLhcPRdMEvolRaFsuTT1/9g8f8vsjoU4bJdHyn+U+Jnh1CDwCjKmgCrXxCKB30ZBEojPwehrwDApBFZ2Iyie/+Uh8bGnLag7A7WnHZAmiMjCO8tpLmyFhEr4eYs1X27Etj0ZJh3HjNfHKyD1J8KOl7z7Er0vscpk/aYSaes5HQkHY8HglFEfn1XR8iSRrtDgtWiwcEqKkJ54OPJwNw0cyvsVi8HDyUQJ/eFb6cgVHZYrEoJCXWUHY4lqiIZkQRZk1fQmCgk6GD9mA2K3gVGUlSiI1u7JZA7roaX/DlSEpKEzl/6kqjFl8ETTdKUu02N26nwOKPwplyRYNRxukLLakKRCV4uOsJQ0/64ttrgVr/uIpi8CUVrW1nzb9ju3wSsOarEDYsCebcKxsYNbF7bb/DYWZg//00NAwhPLy125xTelWxY1dvNm3JZtL4jT6HoHc7vutvgKNV4Tz70sXdPo+B/fcRE1OPeEy3tNmscM8dH/Pa29NYs66zMCAosJ3M9MMArN+Uy+CBexlYsK/be74vNE1k644spkxc59NWUCguSaTpZ1I0k2WF++7+gHc+PPc/uo7vfd6f7Uw9OOPRgogHkRAUPIg0IqPw03xJOpxAxxbVaQ/E5mjHY7FRndiblAPbKSwYiSYK7BhhGGDNbOGpZ66muSUQEPifh+7wV5P8Jzh34lqyMkopK48hNKQdUdSQZUN9a+Xq/pw9ZhsHipOQJZW01EpCQ1oRRZUpE9ezYlV/HvvbdZwzdhOXXvg1kqT7G5eCg9q5986P/KtjowrGyAkIgqEg1oGuRrKrc5g2dQ1tbXZqaoN96mAG8+jCr4cyafwGzFadbasDWb84mIffLMVk1vG4Bd54LI7zrqnvdp2aBuVHoomLq8dsUgmPVnh1ZSG3T0ynuV5GVQwpU2e7wSnVZ4DG/sJkZs+dwLkT1jJowH48XjPzPj+LQQP2+ak2Olb8h49E849/XQ5ASEg7wwbvRtNEH+fRiXeZkRHNPPrgq/zqt7+mY0kQEd7MeZPXHudANA0+WzCGteuNHNDwoTu54pKFyJLmy+PoFOQdwO02UXo45gRn+37I7XuI1tYAXn9nGhPO2UBe9sH/eMyTITamjj5pR4iNqUeWFSaesx4BnXaH7fgGkZ8APQ6hB35IGKGeLFyowF5+XGGPY4v/nLZA7E6jyqYqKZ1ehdsp7D8Sr2wiouYITZFxCIoXXerMDzS3BPmP93i+PfxzMtjtTu6+9WOsVjdJidUIAmSmHwHwk6LJssqa9f0oPpjM+k15gM61Vyzg5us/JSLcIKwbNXwn0VFNKIpRgdNB+qbrhsB8aIixw+raU9C1q1ZVBf/7u6Lj/YYusJu01Cqf4TW6c0cO3+FPND/0WhlrFwZhsepsXxNAv+Ht3PRQJbKpcyzNZ7RtNrfRc+A2YTZ5aXdaqK0088CLpbQ3y/zvPUn+T+nZJ8+iYEogD/3PG4iijtttIiS4jd/d967/Wjr6A1RV8InrGPPLzzlIa5ud0rJ4+uUV43Sa2bYjnRHD9hxzoRrBwQ765RVRVR1BQ2Mwi5cOo7XVznVXf+6noujQW1i6chC6bpx02/ZM+uUVMaBfIbKs+e+lgEZifHdnePdts/li4SgOliSe8nfk6+VDWLR0GG63mV17ehMa+sOKLL4PUnod5bqrPvd/H+Ji6rnhmgUcKD71ef8n6HEIPfAjEI1sX+JXBPp1oahoQURBIPwEcpnfBx3reQE4mpxOStFOFJOZ+phEIqoO0xgVj6B42TFiEgC66Ycb/e+C02mlpCyOSeM3HEfRANDWbkUSNdJ7H2HRkg76BQGP10zfzLIujVoqOX1LcDjNKEpnjb4g4HcGx47d8RwM3qOB/Q2SuuYWGyHBzm7duIoi4nJbCApsx2TS0HQRSdBwu83Ue2XeePd8+qQepmJDCYs+jKS00EpUnIffPHuY2GRv57mBhUuG0ie1gpCQdt6bPZkhA/eQHFvGxEvqGTahFVURWPRRBC6XRPEuO6bQUA6VxDB7zgQumrkMq09BrIPN02RSURSR1evyWbWmH0N93cu6DvO/HM3qdf1wOi0MHriXquoI6upDSU6uJj62zp9fUBURAZ07b5mDLKt8+O/xLF46nJiYBr8zEAQoPNCLjPRy8nIO+qvEXG4Lb78/lcED9uHxSn5HZ7e5Kcg/wL13foDLbWbP3jT69yuitc3+vRyC09mVqVT4WcJF6zfm0dgYzG/vfce3I4R1G3N47a0ZvPrxD++lOFX0JJV7cErYhp2NBHzv43QMJ9AhbemyBaLIJjxmC3WxSbQGhxmhIUFg+6gpxjEmM5psOnHN448EXRf46JOJHKmMPC7hCbBoyTDu/8PdrF5b0O24zz4fw+dfjegyjhEHf+jRm9i+MwM4ftotrXY2bcnqZuQPlsSj6/DFopE89fRVzJ03BlE0OpnffDIWR5voT6YuXTHQHxqTfbuQoCAnEeFtXH7R16iaRFFhHDX2SfzqBS81riw2b0j2zw+g3WFl2/YsXn59Jv/zhztZs66ATxeMJSRC5Y7HjuL1yoiSzt/mFPOPeUWknxXp68sQWLJ8CG6PCa9XQlFF2h1Wfz+CKOrYbG4uu3gJH86ZBBjkcF8vG+ozqAKbtuRQfiQWRZGIiWqkojKabTuMeyXLus+J6j5nMKzjzvL+xxP5459v5u33J1NWHsfmbZlk9Cnrdm/zc4oBnTVr+6EoEsFBDkwmFZNJJS/nIAP67efaK79AVWHE0F386+m/8vgjLyAI/3mo8aeCLBuUHl6v4XSDApzH0ar/VOhxCD34VngQ2I+FfVipwkQTEnuxYuiJfbuxPjbS6bIF4DVb8cpmKlMy0ASBwoJR6JLEjhGTaYxOwBUQZCjN/4RO4FhIkkp0ZDMuV+cuRNfB5TKTlFiDw2EzYrddcP3VCzhvytpu04yLbeCSC5fxwqsX+bn2OxyMwZp5L5VVUQgCNDXbkWWN0sPxPPDwHRypiKGwKIXPF57FHx69lcNFVmb/M4a1C0NYubqAm+58kMED9tPUHMi2HQYlhigazLJ796fw2N+u5ZyxW/nzm/twE8m7H52HLT2P8eeXd2NjDbC7+P39b9O79xFaWgMIC22htCyBPz95Ay6XGYtFQZKgrc3Gn564kbbAYX4eoqSEaoICnaxYNYANG3MICnRRUhbHx5+Mp7XNTr/cIlJTKumbWcKwITt5+ol/+JlZu0JRZN589zz+9PiNPPfSJew/kIwoagb1uKSxZ19vOsJVny0Yx5JlQ6mojGHz1hwWLRlGQV4xgwfsIyionYAAQ3Sp+FAijz55I+98OJW7fnMfio//CKC6JpSGhhDfZ42RGzIpfDz3bH/Y6UxEVGQjO3b14d7f3cNb75+L1eo++UE/Ek5LyEgQhIuAR4C+wJAT6Tb34PRDA/Zj9dFSG9iODQs6iXj4tkRhx7GaICLpGpUpGcQfKqSo/yjaAkMIq66gISbRKIWRfV9B+efv7tR1gflfjiI/t5g+aUeorgknPq6eRUuHsm3HifsVXn7tAh57+AXCw1sQRSO2rWoSr7w5A0lUkSQVl1umsTGYuNgGjh6NQNNEVEXinQ8ms/ybQQwfupP4uDpqasP947aXleFtP8hT25IRBJ03n4glKaOd5sq9fPDxJA4fiSW1VyUF+cWoqoDJpJCZfphn//YMmioAAq88/ziypDF3/hiefu4ybrpuHhHhzX7H8I9/XczO3RmMHL6DC6Yt574HfsXh8mgkX1IWQfdRMER3u+bq2nAefuxGf+3/xs05lFdE8fgjL2GxeHG7ZRRF4r673zdi+ALMnLacqIgmXn7jAqwWDzHRDRwqTfA3lwHExtSzYXOOQQYoKBTkF9LaZiO111F27DJ2EL2SjvLIg6+haQIej4zZrPDsX59GEODBP91C5dFoP01EXEw9JpNGRWUkEeHNxEQbvRQdOz9RBFebhZ27M36079BPgRWrBrFi1SAAvlk9kG9WD/zZzn26cgi7gQuAl0/T+XtwCrCiM5VmFhJCq4+I2oLGeTRjOcYZdNSRqJKMrCq4bHZ0RMxeNzXxaYTWVFEfmwSayq7hE4yDToMT6ApNE/ly0Si8Xpl3PzyXispo+vcrRNfp1vjUFR6viS8Xj+TKSxf6k5jvfDAZRTFKE9esz2fMqO188PFE0lKOcqjMIDn7fOFo/xjrNhxfRmgOtNKw10GtYgMd6qvMNFSbCcqxUHTQCP9kZZRSXRPOqrX5ZGWUkpRYQ4DdjcXia2Tz8RTl5Rxk1+50XE5DrKYjt6GqIpPGr2X40N2EhbYyasR2Zp6/HF2Ht98/l2uv/ByzSSEyoonauk4BIo/H3O1+7NyTDsAf/3wLv733Xb8iHBjnkiSNKROMiqp2x5e0tAYyesQOfv27X2G1eDCZFKxWN4dKEggOasdm86CqAudPWc24s7Zgt7m549f3o+siZeVx/PPFi7n5+k/9OQxdh+devIjKo10dl46iiMz5dBwLvx6OPcDFhdOWMmrkTn//h9ttQtOPV7XrQSdOi0PQdX0f0I0atwdnJkTAgZH4E+hUJoPjS0ddVjuqyYzV0UZlryxiDhexv/8oNEli99BzjDdJZ14dQ0fnKfCtO4OuyMsppuxwLO9/NJkrL/uK/NxiDpfHkphYg+aj0548fgPrNuURGtJ2jOE6HgnxNdz7+Gzuuf0KalatA4zlbFB2NoFpnXTOny8cxYIvR/PXvzxP5dFIXn/nfG68egGBgQ5fSMTYsSQnVvPIg6+h6zB33hj27OvDHbd8zG03zSXA7kZRRFRV5NorvkCSdLZuTyfcJyy/bkMOQUFtaLpAff13d/rW1YcBgk/BTEAUNT9tBRgOatTwHYiSDrrAlInryM8tQkRn644M+maWYrUaXS+SpCNJCmZzK5oK1189n9racLbtyGD/gWTMJqVbJdYVl37FxbOW8MHHk9i5O51LZn2Nw2llwZdnAdDaGsCyVYM5a/QOnC4zVouH5pYA3v9o8kk/3zMVP4e1PPP+O4+BIAg3AzcDJCcnn+bZ/PfBg0AYCkN8mgYdYjRmX7WR22JDUhVMipejKVlEVpZQWDCSprAoghtqqI9NNiyD9MsTC/k2vPrWDFwuM7ou8ucnb8Bi8TB29IT5N8oAAB3fSURBVFYuvmCpX9M3I/0wOdklbNyczd79aSccRxA0REFn2OBdRIS3kJ68j3pJQwoOw9PQhLumxu8Q4uNqmDltBSZZJTConczgdtLSjmC3eXC7ZXRdRZZ1JElHFL04nGZ+//DtNLcEkRhfzdr1eYSGtjJq+C6/0dZ1mPNiBDNvLqJ/vyIEAYYN2cOY0TvYtSeNp5+74jvvg9XqJiSklXc/nExjYxB33fZvv85yR4imo6RW03UunLHUH77asCUb/VtCjpIMI4ftwuOR2bs/lVHDtndrcBNFCAluZ/+BXhwsSUAQdEaN2IHHY2LBl6PpMJ1Bge1s2JTN2x9MJTG+hgumL2fXnj4/4BP/78FP5hAEQVgCxJ7gTw/quj7vVMfRdf0V4BWAQYMG/fSdGT3oBhs6E7uQ3J1DKzrgNlkwe90c7ZVJeM0RAluaOJrUh4DGOuriU0DX2DtorHHQGesMdBITajhS8f2amLqWI2qaiNNp5avFI2hsCuK6Kz8HjPDM0hWD+OiTCd86ziO/f5XkpBpU1eDL+eOTKxD+CouXDuHtl0fh9hHrAbS124mJaiQhvsZf+y9Lqk9ARsJk6uQwAqiujvD3bIiSxjnjtmCzurslwwUBFs2OpHeeh34jWn1OQmDN+jzeeu+877wHVqubS2Z9zR0+xbnn//43f/9G1xLbjvl0iO94PIbJufySxf4/HtvJbNxXuPnsLDLOc9K/4AAARyqiSUqswasYJv/TBWN59q/PIAgaHq8Jk0nh9Rf+gqaJPPSXm9m9tw+79xoOoOhgMk89fc13XlMPfsIqI13Xx+u6nnuCn1N2Bj04M9BROtoBj9lKe3AoqihRkZKB0x5klI7KEvsH+mLlomQkjM/gsGCf3kf404OvEuirWPlPIUsqJpPShdfIy3dt9J9/+SIqKiN9wuxGfmLjlr58On8s5vBwgjI6k58tLYH86fEbaXdYuxlQQTAoLDpW/JoGxYcSiIzojOsfLo/jgYdu94nGwMZlgTx8XRo1FSZyh7bxxG3J3DQmk7eeisFs9lJTE46ifPdasSDvAGNHbyM7q4Tc7IPY7W4qKiNRte5NeB3ocBIWi4LFouDxmFi7Pg9BgOLi+G4lv16P4UCufaCWBx/4iLSUSnQdkhJrEATwuC3Iso7d5uLDf09AUWSsFi8Ws4KiyLz9/lSOVkWeeOI9+E6c8SGjHpweHLsV85os6KKIxe2kIiWTkMYaCgtGoEomCgtGGPv8rq2rZzCSEqv8SVVB0Jk8YR0HipM5WhVBbV34yQf4FkRFNbJuQy7vfnguE87ZQGqvyu98f21dOBs35zD9vG9wuUyYzQpfLxvqU/U6HjExdQQFumhrNxMY4PG/3t5uJSDAxYpVBQwZtA+rxctvHryr27Fhoa3IskZrqxVRFNixxs51I7PQfPxLgiCzpWQ8WZs2kZVZynxfLP5YXHrRItJSKomJMlhW771rtt8RRUU2IUtdVdyMTut9hSn+6igEI58gSyrFhxKRJLDZXP5jyostRCd4WLskkt1HRhO2czvZWaUGh5MGTc2BBAY4+dcrF1BTG86uPX2YNnUVgqgjoNPcEsjqdQUnnHsPTo7TVXY6E3gOiAK+EARhu67rk07HXHrQHfoxj11WOzaXg4qUTIJaGjDVVFKZmoXV2UZtQlp3J3AG7wa64sIZy8jPPYjHIyMIMOGcDUydvJYvFo1gzqfn/OBxP50/zv94wbcY1GMxoGA/R6si+GLRSK64ZCH98wspPmiQ9QmC1q1e/urLFgJgMRv5m45Vd0CAi789ezn79vfm0/ln0zer5DhaD5vNzaq1+Xzw8WRiohq48FfLeO+JYP8Y8aOyKN3RzhO3pxE9vP+3zrf4YCJnn7UFSVL9XEaC0JEvMJrmPvlsHMlJ1QwasA+LRSE7qxTAX5K6d38veiVV0dIaRHbfEkKC2/wOIamPm7pqCy88EI11cDpDBu1F9iWURRECA52oisgdN88F4K33phAc5GDpioFYzF5GjdhJRHjTL1ax7HRD+Dl0On8sDBo0SN+8+dRbFt4Uf/g/938TOjiGOqqGOjiGPCYLjZExRFWVs3bixaQUbqc1LJqaOF9y/2duJPuxIIoaV132JcMG78Zq9eLxyMz5bBxfLxvKz1PL0YmE+BqqqiNQVYnAAAdms5eGRqOZ6rYb57CvMMVfk26zubj1hrlkpB/GavHicpvYV5hCW5uNT+ePo9FHrSAIGvfcMZtX3pxBe7v9hOet37ABd00NtsREnOXl2BITcdXUoCsK0WPHgiAg2e0nrAQcPWI7V176pY+7SUCWNTweEyaTl9Xr+vHmu4ZYUWx0HRdesIycrENYrcZ8N2zK4b3ZU1BVEV0XsFo9PP7IC4SFGpxW1TVh/PHPN9FW2YI5MpIn/vQCW7dnUlMbxjVXfOVPWns8Mp8uGMPylYOIja2n7LBRFpuaUsHh8lhU9UzNW/1wvDHnciT5h12XIAhbdF0fdLL3nfn7+x78pNCP+a3IJprDo9EFgcqUDFRJ5kC/kaiymYPZg6hJTPPlB35aaomfEpom4nJZsFi8aJrR5NXWZufndgYAFZXRfuPV1m73OQMds9nDgIJCxo3ZQoerdjqtbNuRidmkGE1aJoVNm7N5453pfmcARoVTfu5BBvQ78K3ntSclET1uHEHpRk+B88gRUFX65Dq58eI3qFm6FG9T0wmPLcgvRJI1duxKR5YNneSXXp/J3v2p9M3spJaoqomkqDgJs1lBUUTMJoX9hSkoiuzb+RjhqpDgdjweGa9XIjysBV2XsERFIQgCv3/kDuZ8Np5v1gyk/EgMqmo0qOnA18uG4vaY/c4AoKQ04f+lM/i50JND+C9F132hIsm4rXYC2luo7JWBpHgp7Decuphkkop3Up3U20fT2YU+8xeOXslHWbFqAPO/OIsbrplPWmpFty7a0wGr1U1GnzLuvfMjwGikio2p582X/mLoMzxwDwX5BzhaFcHsORO48tKFFOQX+ed9wfRlDB+y2x+Tv/yShUyb+g0HipN49c2Z3c5li4/3P44cNYr6DRvQvV4mXVLLmGnNfLwgF4cU1u2YzIxSmpsD2bE7nc8+H0P5kVgK8vdjtXrYvjOT7Tsz/CywfTNLEEWDT+hQaTwffTKBa6/4gtycgz7mWAPBQe3s3pvGOx+eiyypXHXZVwQFOro5uDGjtrJ5WxZJidVs2tKXyqpIpk9dRZ/e5RQeSPlRP4P/dvSEjP6L0DU0BNAWGEpgWxPlqdmImkL84SLWTLyY+EP7KM/q31m2Ih0vT9mDHx+P/P4Vln0ziMAAJ9OnfoPZbJSSOpwWXn59Jjt3pxMa2kJrawCqKiHLCgF2p7+8NCG+hnvumE1IcJufidTtNvPcSxdTWNTrO86s8eCdzxAe7SEo1MhPeLxmvF4Tny4YwzdrBgDw5KPPU1oWx0uvzzrptTz0wGtIosZfn7kKh9OKrgtIoorZ4j2GRfS7ERtTxxN/epGnn7uUltZA/24gNqaOpqagb03A/3/EzxEy6tkh/JegwwloGE5BESUaouMIaGvmcEY+saWFFOYPRzVZKE/3rZSNTOFpmvF/ByRRJSyshaAgB72Sqxk7aiuvvDmD6VNX4vEYesdVVRHs3G2EdrpSMCuK3E0foqIymtfems5vfvUeHo+EKOrMnjPhJM4AFIeLD5+N5rfPlSOKIMk6CF6OVMZwuDyG6eetxCQrRIS3EBrSxvTzVqKqIkuWD8Hl6jTIkRENXDB9BZKok5RQg67D1Zd/aehM70pn4+acU3YG/fvtp3dqBclJhg7EzGkr2bsvleTEKlat7U9VdU9Z6U+BHofwC4cDAfu3dHx2zQ8IgCpKOAKDCGppoio5HVU2UZQ3FMVk5kif3E7jL/d8LX4uDBm8h5uvm4eqing8MnGxxorYkM8cRVxsPQP778di8XynHnQHBhTsQxB0Vq4ayFmjtzFk0J6TlmFKNhtFnotZuWYF48dtxO02SOReeGUWNpubs8dsJijQgSAYokHTp35D5dEoliwb0m2c6VNXkZ1VQnCQw7+hHDxwL0cqoxg+dDe1teGUlMWfYAbHIzDAyZSJ6/yCQKm9jpLa6yjzvxx1Ssf34Ieh5z//FwwHAp8Ryvk0E9SldayjWqgjyNMWFE5QawNVSWkoJiuBLc2UZRYQVX6Qoyk+7p6encBpwboNeZhkhSsvW4jZrKCqArV1IbzzwRR27zV2BSnJlX5a7ZNh6/a+rFrTnyOVMSxcMpzkpKqTHiMIApLFwsD++2luDmLLtizOGbeJgvwivlnTnwcevp2//uU5AuwGDXPp4Tgee+o6VB9vU2CAA0HUGVBQiK4b84+OMnIJrW12ysriSIyvZerk1WzdnkltfRhFxd9NQ7NqbX9aWgO47aZPsJgVXG4Tc+d1VIL14KdCj0P4BcKJgBORcsyAQBEWeuHBhobNty9oDYkguLkeVRCpi0smsLWR0qyBRFaWGrsCs4WjqVnGgD35gdMIge27Mrj2yi98Yu4q9fUhfmcAUHr41FbVQLfwUENjiL+E9VQwe84Edu/pjcttYd2mXH84yGrxEGB343BYMFu8xEbXo+k+rYTEKh79w6tomoDbIyMK+NlPPR5DsGbYkN2GrGZuMQP7F/LNmoKTOgSAuNg6LGaFhsYgwsNaSUyoPuVr+f+InyPd2+MQfoHYip0yLEg+41+Elf3YSMDDWbShCQK18SkENTdQk5SG12yhOHcIitlCVXJ6527gjOUYOrMhiSqjR25nxaofh6c+O7MEp9PCa29PIzf7EKOG7/CRuf28jnrz1mz/40NdpCaDAh1s3NKX9z6cQmCgg8svXozV4sHpslJ+JJZ//OsSbr1xLjYfPbWiiLzy5gx27+3Nr+98n169jF2Kqkp8sWgE8z4fc8pz+uiTc1i0ZDhDBu6md2rFj3SlPfg29DiEXyCG044JnRI6E3opuMkMsUJzG7UJqXjMVg7mDKI6MY2w6iPUJvlYHnucwH+MvlmlXHPFlxwoTqbyaNR/PN6W7Vns2JWO02Vl244s5n1x1s/uDL4LZeVxvPjqhQC0tgXw9392Z0HdsSsdxSujSIYUpsNhZdOWbEDA6bIiSxqKImKxeHxSoKd2bQu/7pQq3bA5jw2b877j3T34MdDjEH5h0AFdEAjTVYp9r6mAKTiI+sRkwprrOZg9mJC6o1QkGyu+2gQf/XJPaOg/wugR24iJaSA7q8So8794EaWH4ygpiWfL9r4/eFyv14TX25kjaGkJ/DGm+7MhPq4Om83FZwvGIIo6089byTVXfIGuC2Sml6HrsHNPbyLDm+md0rPKP5PR4xB+ATi2f6A1NJKqRgcRKET16UN9cTFHzFFY7IEczBmM12I1KKjBcAI9u4IfBRHhzUydtNb/PDurhL6Zpbz9/rmnb1JnAKqrw/ntH+7y5ytWr8vn0gu/ZvDA/f73FOQV4XJZeOu980/XNHtwCugpLTnD0eEEVEH0P69K6s1QHBTEhCKFhZHSN4OQwYNpiEmkIs0XBxbFnh3Bj4zPPh/Le7Mn4XIZq3mPR+b5ly/yN26d6bj+6vn0SSv/0cdVNalb8rqhMZQXXr2IVWv7GQynAAg8+KfbTrnstAcnwM/w/9zjEM5Q6IAidH48LWFGI05DTCJOexDlfQdQ3G84uihR0ScXUZY7nUCPI/jJkJRYjdXqpb4hGItFITam7uQHnQEIsDsZOWwn48aceqf/f4o+aeUIAtTVhSCKOgnxNT/buXvww9ATMjqDcFxoKCyS0IYaGqLiqU5IJaShlgP5w7G2t1LeJxeApihf52ePE/hZ4HKZefmN6azfmMfEc9b7kqRnLoYO2s0lFy7BJCsoqsiQgXvJziqhrc3OI4/fdMpEcLKscN1VC3jjnWmnfMyh0gReev0CKiqjmXH+Cr/YfQ/OXPQ4hDMEfrZRQUTUNQSgslcGIQ21FBaMxOx0UJLVH4/VjsdiM97csxv4SdG/XyHbd6Z30ySYPadTtmPx0uGnY1rfC7v39mZk5Q76ZpQiy4ZeQYDdxb/nnvO9WEFz+h5ixNDdrNuQ55elPBlee2uG//Enn/Xwiv0S0BMyOo3o6ChWBNHfZ9waFokONEbE4ggIoTSrAI/VTltoBOXpvrK7nvzAT46IiCbuvu1jsjLKTv7mMxjtDhsvvjILUdT9esYbNuWydkO/Uzr+wplLePj/2rv32KrP+47j74/vdwyGcYkd7hhSrpFDN9RlScgStnUhu0mduqVds2X5o2o2aeqWsTVqu2irMk2Vtn9ClUitRndTEg1FWUNII3XdRhJIIYRAEGsxl0YECDfb+HLs7/44P4PDAr7g48eXz0uyOMf8fPwBwfn4+f2e3/M8/i1+73deJAIe/tx2nnj8W/z6A98vZGxLxCOEMTbwInFx5GvgUn0DNRfOcrF+JicXLKf2w9McWvcpSrq7ODZwoTkruGVLWmlqPMXiRSeIgF/9pR8yb+5pzpytZ9/+ZYO/wDi0auURioqC/3l9Fbet+DFrVx/mo4ubXN+7hxZxz517KCvrQcrfpFZWluPg8wsLntvGngthDHzcmdO2rASK+/o4sWgFzT/6Lw7dfifK5TjavIbuymq6K7LdrjwaGDMLF5zkM7/5Kr29QoJlS1tZsfwor/3g9glbCK3H5vLXf/sQh4/Mp6y0h3Vr3mOoN4e9e3ARz3z7AR55+AWCPvr6itj67IMcfM+FMBn5x84CG/hzWK+KyGWbzBxf9AlyJaVcnD6L9roGji9ZRVdlNZ01dZxYlg3nfWpozL28cwNPP/MguVz+/HpPTwnPb/8FvvPdX0mcbOROfdDA4SP5NY66e0p5fffKYX392jWHKSnuY89bKygp7ssKxSYjjxAKYOBsoT5Eb2kZpT1dtNXPoLyjg8s1dVyqb+DEwhWcblxMRB/HmweUgCVVXX2Z8vIcnZ1lVFR0U1fXnjpSUvvfWczO19bTemwuL+/8KQ0N51JHmprGYHU7F8Io6y+BouxEUdu0Bopz3XRVVHF8yUqW7H+DAy13gXR1JNDPo4FxYVpdOzteXc+/vbCRe+96k6bGqb3K5ht7ro4oftI6zzeXTWIuhFFwbW+31U2ntOsylV2XObZ0JQsP/ogDd9xNrqSEEwuW01U1YK0al8C48/z2u688/t7O8T+11Gy0JDk/IekpSYckvS3pBUn1KXLcrP5po5AfFXRWVANwbOkqesoraaup53zDbN6/dSmd1bXkyis52T911PcQmNk4k+qE9SvAyohYDRwGHk+UY9gGbkvZB3RkP+2319bTWVlNe3Ud52bN41TTYg6svwcQJ5dmQ+7+EnARmNk4lKQQImJHROSyp7uAxhsdP170l0D/TWQdNdPorKrlcnUtrctWExIHPrkxf5fx4k/QWV1Lb1m2Z4FLwMxuwli8hYyHawhfAP4ldYjruXaj+ss1dRCiuv0CR5etYc7x/2X/+o10VVZR1tXJ5eq6q1/sEjCzCaRghSBpJzDnY35rS0T8e3bMFiAHbLvB6zwCPAJw662D78M6WvpHA0XZ47ba6dReOkfrsjU0nGylqC/Hh7MbKcl1c7kmv/Tv+4tW5KeGuQjMbAIqWCFExL03+n1Jnwc+DWyMuP4E24jYCmwFaGlpKehE3IH3DwT5KaN1F85yuaqW9rrplPZ0cWbOrdDbR+tttxNF4tSC5R8tAZeBmU1QqWYZbQK+DDwQER0pMlyrvwR6s+edlTVcqp9BV3klR5vX0iex/2fvI1TE6flL86OCoqxPXQJmNgmkuobwD0A58Iryb6a7IuLRsQ5xZaE5iiimj66KKjorqqg/f4ajzWupOXeat3/uPjqqavMzimqv7grlEjCzySZJIUTE0BZUL9T3B3oliiMI4OKMWUz/8BRHm9dRefEcVR3tfDBvAb1FRXTU5m+RONu02NcHzGxSmxIL5wy8gaz/1wszZgPQXV7JhYbZ9JSWc6pxERcaZrNvw31QJM42Lsof7OsDZjYFTIlC6Ne/R3F3aTnnZs0jV1LGT5rX0Sexb8P9IHFu3vz8qKB/P2OXgJlNEZO+EHqKsmWMS8o4P3MuAEeXryMk9m64n1NNi7lUP5P2uukfHQm4CMxsipnUhRAwoATWcvZn5pIrKeX9+ctoq5tB+7QZUFTE+TlN+S9wCZjZFDapC6GnpJQzsxvJlZTy0wXLaaubwd4Nm0Diwuxb8gf5+oCZGTA+lq4omNbmtURREfuyErg0a95HN5lwCZiZXTGpRwgnF91GR8002qbNuPpJXx8wswmo8PulTfJCQKKtYfaVx2Zmdn2TuxD6uQzMzAY1NQrBzMwG5UIwMzPAhWBmZhkXgpmZAS4EMzPLuBDMzAxwIZiZWcaFYGZmgAvBzMwyLgQzMwNcCGZmlnEhmJlNABqDNdlcCGZmBiQqBElfl/S2pL2SdkialyKHmZldlWqE8FRErI6ItcCLwFcS5TAzs0ySQoiIiwOeVjM2mwGZmdkNJNtTWdKTwEPABeDuVDnMzCyvYCMESTslvfMxH5sBImJLRDQB24Av3uB1HpG0W9Lu06dPFyqumdmUV7ARQkTcO8RDtwEvAU9c53W2AlsBWlpafGrJzKxAUs0yWjrg6WbgUIocZmZ2VaprCH8jqRnoA1qBRxPlMDOzTJJCiIjfGIvvI0H4JJOZ2ZBM6juV59xSlzqCmdlNq6wsobi48G/Xk7oQPveHn6S4uPDrf5iZFUpxsfjs798xJt9rUhfCilVz+OO/uIe5jdNSRzEzG7YZM6v4g8c28PMbl4zJ91NMoJPsLS0tsXv37tQxzMwmFEl7IqJlsOMm9QjBzMyGzoVgZmaAC8HMzDIuBDMzA1wIZmaWcSGYmRkwwaadSjpNfu2j0TYTOFOA1x0rzp+W86fl/IObHxGzBjtoQhVCoUjaPZQ5uuOV86fl/Gk5/+jxKSMzMwNcCGZmlnEh5G1NHeAmOX9azp+W848SX0MwMzPAIwQzM8u4EDKSvi7pbUl7Je2QNC91puGQ9JSkQ9mf4QVJ9akzDYek35J0QFKfpHEx42IoJG2S9J6kI5L+LHWe4ZD0rKQPJL2TOstwSWqS9Jqkd7N/N4+lzjQckiokvSFpX5b/q6kzgU8ZXSGpLiIuZo+/BNwWERNmr2dJ9wHfj4icpG8ARMSfJo41ZJJWkN9j+2ngTyJi3K9zLqkYOAz8InACeBP47Yh4N2mwIZJ0J9AGfCciVqbOMxyS5gJzI+ItSbXAHuDBCfR3L6A6ItoklQI/BB6LiF0pc3mEkOkvg0w1MKGaMiJ2REQue7oLaEyZZ7gi4mBEvJc6xzCtB45ExI8johv4Z2Bz4kxDFhE/AD5MnWMkIuL9iHgre3wJOAjckjbV0EVeW/a0NPtI/p7jQhhA0pOSjgOfBb6SOs9N+ALwH6lDTAG3AMcHPD/BBHpTmiwkLQDWAa+nTTI8kool7QU+AF6JiOT5p1QhSNop6Z2P+dgMEBFbIqIJ2AZ8MW3a/2+w/NkxW4Ac+T/DuDKU/GbDIakGeA74o2tG+eNeRPRGxFryo/n1kpKftitJHWAsRcS9Qzx0G/AS8EQB4wzbYPklfR74NLAxxuHFoWH8/U8UJ4GmAc8bs8/ZGMjOvT8HbIuI51PnGamIOC/pNWATkPQC/5QaIdyIpKUDnm4GDqXKMhKSNgFfBh6IiI7UeaaIN4GlkhZKKgM+A2xPnGlKyC7KPgMcjIi/S51nuCTN6p8JKKmS/MSE5O85nmWUkfQc0Ex+pksr8GhETJif9iQdAcqBs9mndk2wWVK/Bvw9MAs4D+yNiPvTphqcpF8GvgkUA89GxJOJIw2ZpH8C7iK/2uYp4ImIeCZpqCGS9CngP4H95P/PAvx5RLyULtXQSVoNfJv8v5si4F8j4mtpU7kQzMws41NGZmYGuBDMzCzjQjAzM8CFYGZmGReCmZkBLgSzUSPpe5LOS3oxdRazkXAhmI2ep4DfTR3CbKRcCGbDJOmObN+JCknV2Xr2KyPiVeBS6nxmIzWl1jIyGw0R8aak7cBfAZXAP0bEhNtkxuxaLgSzkfka+bWMOoEvJc5iNip8yshsZBqAGqAWqEicxWxUuBDMRuZp4C/JL5X+jcRZzEaFTxmZDZOkh4CeiPhutq/yf0u6B/gqsByokXQCeDgiXk6Z1Ww4vNqpmZkBPmVkZmYZF4KZmQEuBDMzy7gQzMwMcCGYmVnGhWBmZoALwczMMi4EMzMD4P8A9tRNdSrQ0B0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "# 均匀生成40000个数据点\n",
    "x1, x2 = paddle.meshgrid(paddle.linspace(-math.pi, math.pi, 200), paddle.linspace(-math.pi, math.pi, 200))\n",
    "x = paddle.stack([paddle.flatten(x1), paddle.flatten(x2)], axis=1)\n",
    "\n",
    "# 预测对应类别\n",
    "y = runner.predict(x)\n",
    "y = paddle.squeeze(paddle.cast((y>=0.5),dtype='float32'),axis=-1)\n",
    "\n",
    "# 绘制类别区域\n",
    "plt.ylabel('x2')\n",
    "plt.xlabel('x1')\n",
    "plt.scatter(x[:,0].tolist(), x[:,1].tolist(), c=y.tolist(), cmap=plt.cm.Spectral)\n",
    "\n",
    "plt.scatter(X_train[:, 0].tolist(), X_train[:, 1].tolist(), marker='*', c=paddle.squeeze(y_train,axis=-1).tolist())\n",
    "plt.scatter(X_dev[:, 0].tolist(), X_dev[:, 1].tolist(), marker='*', c=paddle.squeeze(y_dev,axis=-1).tolist())\n",
    "plt.scatter(X_test[:, 0].tolist(), X_test[:, 1].tolist(), marker='*', c=paddle.squeeze(y_test,axis=-1).tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.3 自动梯度计算和预定义算子\n",
    "\n",
    "虽然我们能够通过模块化的方式比较好地对神经网络进行组装，但是每个模块的梯度计算过程仍然十分繁琐且容易出错。在深度学习框架中，已经封装了自动梯度计算的功能，我们只需要聚焦模型架构，不再需要耗费精力进行计算梯度。\n",
    "\n",
    "飞桨提供了`paddle.nn.Layer`类，来方便快速的实现自己的层和模型。模型和层都可以基于`paddle.nn.Layer`扩充实现，模型只是一种特殊的层。\n",
    "\n",
    "继承了`paddle.nn.Layer`类的算子中，可以在内部直接调用其它继承`paddle.nn.Layer`类的算子，飞桨框架会自动识别算子中内嵌的`paddle.nn.Layer`类算子，并自动计算它们的梯度，并在优化时更新它们的参数。\n",
    "\n",
    "### 4.3.1 利用预定义算子重新实现前馈神经网络\n",
    "下面我们使用Paddle的预定义算子来重新实现二分类任务。\n",
    "主要使用到的预定义算子为`paddle.nn.Linear`：\n",
    "\n",
    "`class paddle.nn.Linear(in_features, out_features, weight_attr=None, bias_attr=None, name=None)`\n",
    "\n",
    "`paddle.nn.Linear`算子可以接受一个形状为[batch_size,∗,in_features]的**输入张量**，其中\"∗\"表示张量中可以有任意的其它额外维度，并计算它与形状为[in_features, out_features]的**权重矩阵**的乘积，然后生成形状为[batch_size,∗,out_features]的**输出张量**。 `paddle.nn.Linear`算子默认有偏置参数，可以通过`bias_attr=False`设置不带偏置。\n",
    "\n",
    "***\n",
    "paddle.nn 目录下包含飞桨框架支持的神经网络层和相关函数的相关API。paddle.nn.functional下都是一些函数实现。\n",
    "***\n",
    "\n",
    "\n",
    "代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.nn as nn\r\n",
    "import paddle.nn.functional as F\r\n",
    "from paddle.nn.initializer import Constant, Normal, Uniform\r\n",
    "\r\n",
    "class Model_MLP_L2_V2(paddle.nn.Layer):\r\n",
    "    def __init__(self, input_size, hidden_size, output_size):\r\n",
    "        super(Model_MLP_L2_V2, self).__init__()\r\n",
    "        # 使用'paddle.nn.Linear'定义线性层。\r\n",
    "        # 其中第一个参数（in_features）为线性层输入维度；第二个参数（out_features）为线性层输出维度\r\n",
    "        # weight_attr为权重参数属性，这里使用'paddle.nn.initializer.Normal'进行随机高斯分布初始化\r\n",
    "        # bias_attr为偏置参数属性，这里使用'paddle.nn.initializer.Constant'进行常量初始化\r\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size,\r\n",
    "                                weight_attr=paddle.ParamAttr(initializer=Normal(mean=0., std=1.)),\r\n",
    "                                bias_attr=paddle.ParamAttr(initializer=Constant(value=0.0)))\r\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size,\r\n",
    "                                weight_attr=paddle.ParamAttr(initializer=Normal(mean=0., std=1.)),\r\n",
    "                                bias_attr=paddle.ParamAttr(initializer=Constant(value=0.0)))\r\n",
    "        # 使用'paddle.nn.functional.sigmoid'定义 Logistic 激活函数\r\n",
    "        self.act_fn = F.sigmoid\r\n",
    "        \r\n",
    "    # 前向计算\r\n",
    "    def forward(self, inputs):\r\n",
    "        z1 = self.fc1(inputs)\r\n",
    "        a1 = self.act_fn(z1)\r\n",
    "        z2 = self.fc2(a1)\r\n",
    "        a2 = self.act_fn(z2)\r\n",
    "        return a2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.3.2 完善Runner类\n",
    "\n",
    "基于上一节实现的 `RunnerV2_1` 类，本节的 RunnerV2_2 类在训练过程中使用自动梯度计算；模型保存时，使用`state_dict`方法获取模型参数；模型加载时，使用`set_state_dict`方法加载模型参数."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class RunnerV2_2(object):\r\n",
    "    def __init__(self, model, optimizer, metric, loss_fn, **kwargs):\r\n",
    "        self.model = model\r\n",
    "        self.optimizer = optimizer\r\n",
    "        self.loss_fn = loss_fn\r\n",
    "        self.metric = metric\r\n",
    "\r\n",
    "        # 记录训练过程中的评估指标变化情况\r\n",
    "        self.train_scores = []\r\n",
    "        self.dev_scores = []\r\n",
    "\r\n",
    "        # 记录训练过程中的评价指标变化情况\r\n",
    "        self.train_loss = []\r\n",
    "        self.dev_loss = []\r\n",
    "\r\n",
    "    def train(self, train_set, dev_set, **kwargs):\r\n",
    "        # 将模型切换为训练模式\r\n",
    "        self.model.train()\r\n",
    "\r\n",
    "        # 传入训练轮数，如果没有传入值则默认为0\r\n",
    "        num_epochs = kwargs.get(\"num_epochs\", 0)\r\n",
    "        # 传入log打印频率，如果没有传入值则默认为100\r\n",
    "        log_epochs = kwargs.get(\"log_epochs\", 100)\r\n",
    "        # 传入模型保存路径，如果没有传入值则默认为\"best_model.pdparams\"\r\n",
    "        save_path = kwargs.get(\"save_path\", \"best_model.pdparams\")\r\n",
    "\r\n",
    "        # log打印函数，如果没有传入则默认为\"None\"\r\n",
    "        custom_print_log = kwargs.get(\"custom_print_log\", None) \r\n",
    "        \r\n",
    "        # 记录全局最优指标\r\n",
    "        best_score = 0\r\n",
    "        # 进行num_epochs轮训练\r\n",
    "        for epoch in range(num_epochs):\r\n",
    "            X, y = train_set\r\n",
    "            # 获取模型预测\r\n",
    "            logits = self.model(X)\r\n",
    "            # 计算交叉熵损失\r\n",
    "            trn_loss = self.loss_fn(logits, y)\r\n",
    "            self.train_loss.append(trn_loss.item())\r\n",
    "            # 计算评估指标\r\n",
    "            trn_score = self.metric(logits, y).item()\r\n",
    "            self.train_scores.append(trn_score)\r\n",
    "\r\n",
    "            # 自动计算参数梯度\r\n",
    "            trn_loss.backward()\r\n",
    "            if custom_print_log is not None:\r\n",
    "                # 打印每一层的梯度\r\n",
    "                custom_print_log(self)\r\n",
    "\r\n",
    "            # 参数更新\r\n",
    "            self.optimizer.step()\r\n",
    "            # 清空梯度\r\n",
    "            self.optimizer.clear_grad()\r\n",
    "\r\n",
    "            dev_score, dev_loss = self.evaluate(dev_set)\r\n",
    "            # 如果当前指标为最优指标，保存该模型\r\n",
    "            if dev_score > best_score:\r\n",
    "                self.save_model(save_path)\r\n",
    "                print(f\"[Evaluate] best accuracy performence has been updated: {best_score:.5f} --> {dev_score:.5f}\")\r\n",
    "                best_score = dev_score\r\n",
    "\r\n",
    "            if log_epochs and epoch % log_epochs == 0:\r\n",
    "                print(f\"[Train] epoch: {epoch}/{num_epochs}, loss: {trn_loss.item()}\")\r\n",
    "                \r\n",
    "    # 模型评估阶段，使用'paddle.no_grad()'控制不计算和存储梯度\r\n",
    "    @paddle.no_grad()\r\n",
    "    def evaluate(self, data_set):\r\n",
    "        # 将模型切换为评估模式\r\n",
    "        self.model.eval()\r\n",
    "\r\n",
    "        X, y = data_set\r\n",
    "        # 计算模型输出\r\n",
    "        logits = self.model(X)\r\n",
    "        # 计算损失函数\r\n",
    "        loss = self.loss_fn(logits, y).item()\r\n",
    "        self.dev_loss.append(loss)\r\n",
    "        # 计算评估指标\r\n",
    "        score = self.metric(logits, y).item()\r\n",
    "        self.dev_scores.append(score)\r\n",
    "        return score, loss\r\n",
    "    \r\n",
    "    # 模型测试阶段，使用'paddle.no_grad()'控制不计算和存储梯度\r\n",
    "    @paddle.no_grad()\r\n",
    "    def predict(self, X):\r\n",
    "        # 将模型切换为评估模式\r\n",
    "        self.model.eval()\r\n",
    "        return self.model(X)\r\n",
    "\r\n",
    "    # 使用'model.state_dict()'获取模型参数，并进行保存\r\n",
    "    def save_model(self, saved_path):\r\n",
    "        paddle.save(self.model.state_dict(), saved_path)\r\n",
    "\r\n",
    "    # 使用'model.set_state_dict'加载模型参数\r\n",
    "    def load_model(self, model_path):\r\n",
    "        state_dict = paddle.load(model_path)\r\n",
    "        self.model.set_state_dict(state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.3.3 模型训练\n",
    "\n",
    "实例化RunnerV2类，并传入训练配置，代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.37500\n",
      "[Train] epoch: 0/1000, loss: 0.809099555015564\n",
      "[Evaluate] best accuracy performence has been updated: 0.37500 --> 0.39375\n",
      "[Evaluate] best accuracy performence has been updated: 0.39375 --> 0.41875\n",
      "[Evaluate] best accuracy performence has been updated: 0.41875 --> 0.44375\n",
      "[Evaluate] best accuracy performence has been updated: 0.44375 --> 0.45000\n",
      "[Evaluate] best accuracy performence has been updated: 0.45000 --> 0.45625\n",
      "[Evaluate] best accuracy performence has been updated: 0.45625 --> 0.46875\n",
      "[Evaluate] best accuracy performence has been updated: 0.46875 --> 0.47500\n",
      "[Evaluate] best accuracy performence has been updated: 0.47500 --> 0.49375\n",
      "[Evaluate] best accuracy performence has been updated: 0.49375 --> 0.50625\n",
      "[Evaluate] best accuracy performence has been updated: 0.50625 --> 0.51875\n",
      "[Evaluate] best accuracy performence has been updated: 0.51875 --> 0.52500\n",
      "[Evaluate] best accuracy performence has been updated: 0.52500 --> 0.53750\n",
      "[Evaluate] best accuracy performence has been updated: 0.53750 --> 0.55625\n",
      "[Evaluate] best accuracy performence has been updated: 0.55625 --> 0.57500\n",
      "[Evaluate] best accuracy performence has been updated: 0.57500 --> 0.59375\n",
      "[Evaluate] best accuracy performence has been updated: 0.59375 --> 0.61250\n",
      "[Evaluate] best accuracy performence has been updated: 0.61250 --> 0.62500\n",
      "[Evaluate] best accuracy performence has been updated: 0.62500 --> 0.63125\n",
      "[Evaluate] best accuracy performence has been updated: 0.63125 --> 0.63750\n",
      "[Evaluate] best accuracy performence has been updated: 0.63750 --> 0.65000\n",
      "[Evaluate] best accuracy performence has been updated: 0.65000 --> 0.65625\n",
      "[Evaluate] best accuracy performence has been updated: 0.65625 --> 0.66250\n",
      "[Evaluate] best accuracy performence has been updated: 0.66250 --> 0.68125\n",
      "[Evaluate] best accuracy performence has been updated: 0.68125 --> 0.68750\n",
      "[Evaluate] best accuracy performence has been updated: 0.68750 --> 0.70000\n",
      "[Evaluate] best accuracy performence has been updated: 0.70000 --> 0.70625\n",
      "[Evaluate] best accuracy performence has been updated: 0.70625 --> 0.71250\n",
      "[Evaluate] best accuracy performence has been updated: 0.71250 --> 0.71875\n",
      "[Evaluate] best accuracy performence has been updated: 0.71875 --> 0.72500\n",
      "[Evaluate] best accuracy performence has been updated: 0.72500 --> 0.73125\n",
      "[Train] epoch: 50/1000, loss: 0.5660088658332825\n",
      "[Evaluate] best accuracy performence has been updated: 0.73125 --> 0.73750\n",
      "[Evaluate] best accuracy performence has been updated: 0.73750 --> 0.75000\n",
      "[Evaluate] best accuracy performence has been updated: 0.75000 --> 0.75625\n",
      "[Evaluate] best accuracy performence has been updated: 0.75625 --> 0.76875\n",
      "[Evaluate] best accuracy performence has been updated: 0.76875 --> 0.77500\n",
      "[Evaluate] best accuracy performence has been updated: 0.77500 --> 0.78125\n",
      "[Evaluate] best accuracy performence has been updated: 0.78125 --> 0.78750\n",
      "[Evaluate] best accuracy performence has been updated: 0.78750 --> 0.79375\n",
      "[Evaluate] best accuracy performence has been updated: 0.79375 --> 0.80000\n",
      "[Evaluate] best accuracy performence has been updated: 0.80000 --> 0.80625\n",
      "[Train] epoch: 100/1000, loss: 0.49134373664855957\n",
      "[Evaluate] best accuracy performence has been updated: 0.80625 --> 0.81250\n",
      "[Evaluate] best accuracy performence has been updated: 0.81250 --> 0.81875\n",
      "[Evaluate] best accuracy performence has been updated: 0.81875 --> 0.82500\n",
      "[Evaluate] best accuracy performence has been updated: 0.82500 --> 0.83125\n",
      "[Evaluate] best accuracy performence has been updated: 0.83125 --> 0.83750\n",
      "[Train] epoch: 150/1000, loss: 0.45692673325538635\n",
      "[Train] epoch: 200/1000, loss: 0.439577579498291\n",
      "[Train] epoch: 250/1000, loss: 0.43059825897216797\n",
      "[Evaluate] best accuracy performence has been updated: 0.83750 --> 0.84375\n",
      "[Train] epoch: 300/1000, loss: 0.425873339176178\n",
      "[Evaluate] best accuracy performence has been updated: 0.84375 --> 0.85000\n",
      "[Train] epoch: 350/1000, loss: 0.4233371615409851\n",
      "[Evaluate] best accuracy performence has been updated: 0.85000 --> 0.85625\n",
      "[Train] epoch: 400/1000, loss: 0.42193928360939026\n",
      "[Evaluate] best accuracy performence has been updated: 0.85625 --> 0.86250\n",
      "[Train] epoch: 450/1000, loss: 0.42114076018333435\n",
      "[Train] epoch: 500/1000, loss: 0.42066121101379395\n",
      "[Train] epoch: 550/1000, loss: 0.42035332322120667\n",
      "[Train] epoch: 600/1000, loss: 0.4201384484767914\n",
      "[Train] epoch: 650/1000, loss: 0.41997450590133667\n",
      "[Train] epoch: 700/1000, loss: 0.41983866691589355\n",
      "[Train] epoch: 750/1000, loss: 0.41971856355667114\n",
      "[Train] epoch: 800/1000, loss: 0.41960734128952026\n",
      "[Train] epoch: 850/1000, loss: 0.419501394033432\n",
      "[Train] epoch: 900/1000, loss: 0.4193987250328064\n",
      "[Train] epoch: 950/1000, loss: 0.4192982316017151\n"
     ]
    }
   ],
   "source": [
    "# 设置模型\r\n",
    "input_size = 2\r\n",
    "hidden_size = 5\r\n",
    "output_size = 1\r\n",
    "model = Model_MLP_L2_V2(input_size=input_size, hidden_size=hidden_size, output_size=output_size)\r\n",
    "\r\n",
    "# 设置损失函数\r\n",
    "loss_fn = F.binary_cross_entropy\r\n",
    "\r\n",
    "# 设置优化器\r\n",
    "learning_rate = 0.2\r\n",
    "optimizer = paddle.optimizer.SGD(learning_rate=learning_rate, parameters=model.parameters())\r\n",
    "\r\n",
    "# 设置评价指标\r\n",
    "metric = accuracy\r\n",
    "\r\n",
    "# 其他参数\r\n",
    "epoch_num = 1000\r\n",
    "saved_path = 'best_model.pdparams'\r\n",
    "\r\n",
    "# 实例化RunnerV2类，并传入训练配置\r\n",
    "runner = RunnerV2_2(model, optimizer, metric, loss_fn)\r\n",
    "\r\n",
    "runner.train([X_train, y_train], [X_dev, y_dev], num_epochs=epoch_num, log_epochs=50, save_path=\"best_model.pdparams\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "将训练过程中训练集与验证集的准确率变化情况进行可视化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAFDCAYAAACDcoJmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XeYZGWZ///3fU5VdXWenpwDMIk44DBkAZchuICIC79RdxUXZMXFgHz9Er6L62JCVwREcEVFzAhGWJEkikicARGYGcLMAMPkHDpVV9W5f39UddPTPaF7prqqu/rzuq66uup5TrgZrj591xPN3RERERGR/i8odQAiIiIi0jNK3EREREQGCCVuIiIiIgOEEjcRERGRAUKJm4iIiMgAocRNREREZIBQ4iYiIiIyQChxExERERkglLiJiIiIDBCxUgfQF4YPH+6TJ08udRgiUkTPPvvsBncfUeo4CkHPMJHBpTfPr7JM3CZPnsyCBQtKHYaIFJGZvVnqGApFzzCRwaU3zy91lYqIiIgMEErcRERERAYIJW4iIiIiA4QSNxEREZEBQombiIiIyABRlrNKZXCKoogVK1bQ1NRU6lCkwOLxOCNHjqSurq7UoYiIlJQSNykbGzZswMyYPn06QaDG5HLh7rS0tLBy5UoAJW8iMqjpr5uUjS1btjBq1CglbWXGzKiqqmLcuHGsW7eu1OGIiJSU/sJJ2chms8Tj8VKHIX2ksrKSdDpd6jBEREpKXaVSVsys1CFIH9H/WxmI0q+vx9syO5RZdQXx8UNz9UvX4ZnsDvVBbZLY2AYA2l5bC1G0Y319JbHRQ3L1r64B9x3rh1QTG1WHR076tTXdYgqH1hCOqMUzWdJLu7dih8NrCYfV4G0Z0q+v714/so6woZqoNU3mzQ3d60fXE9ZXETW3kXlrY7f62NgGgtokUWOKzMpN3evHDyWoriDa3kJm1Zbu9ROHEVQmyG5tJrtma7f6+OThWEWc7OYmsuu2da/fbyQWD8lubCS7YXv3+gNGYWFAdv12spsau9dPG42ZkVm7lWhL846VZiSmje52TiEN+sQt9fxyACpmTSxxJCIiUg5Sf19O0FBNfOIwUs8v7/bHPRzX0JG4tT77Bt6U2qE+Nnl4R+LW+swy6JL4xaeO6kjcWp9Y0i1xS8wcS2xUHUSeq+8iceiEjsRtZ/UV75icS9xS6Z3WJ4/en7ChGm9O7bz+hGm5xG17y07rK981M5e4bWnaaX3VqQcTVFeQ3dC48/qG6lzitnbbTuvDkXWEFXEyq7aQemZZt/rYuAYsHpJZsYnUc913mopPHg5hQPrNDbS9uKJ7/dRRYEZ62XrSL6/ucvO+T9zMu/wPLwezZ8/2nu7z13T/C5B1qv/xsD6OSvra4sWLmTlzZqnDkD60u//HZvasu88uckh9ojfPMOl/tv3kCRIHjCJ59P5ELW0Qdfk7GwYEydywjqi5rVvitWN9CrpUWyzAKvL1XZK+XH2IVcRwd7y5rXt9PMQSMTxyvGUn9YkQi++uPobFQzyK8JbuwxesIobFQjwb4a27qc9EeGpn9XEsFuCZLJ7KdK9PxrEwwNPZbq2ZAFYZx4Ld1SewwPB0Bm/Ldq+vSmBmeFsGT++mPpXp1loKEFRXdCvbk948vwZ9i5sFAZHGzUgZe/nll5k5cybz589n9uy9z2vuv/9+zjjjDNavX8/w4cMLGKFI/+ORE21qBDPCYTUAZDc1dftDbvGQcGh1rn5jY+4PeTqLJXN/XoPKxG7vE1TtqX73ScDukgQzw3ZXH+xrfbD7+nAP9bEAi+2uPsRi4a7r4yEW35f6GBbfdRpkiRiW2E19RQyrKH4aNegTN8IAMtGejxPpI3sauzVp0iTeeOONvb7+1KlTWb16tZItkV7IvL6elr+8QjC0mpr3HAFAy+OvEXUZExWOqqP63bkem5ZHXyba2gKA7SHhEtlbStzCoHsztkgRrV799hiJJ554gve9730899xzjBkzBoAw3Pk3xra2NhKJ3X9bbz9/9Oi+HXMhUm7auyCTR+3fUZY8ej/o0rXWuUUmedzUXENAYIQjtd6g9I1BvxxI5u9riTa0lDoMGcRGjx7d8Ro6NDdgecSIER1lI0aM6Djuv/7rv7j44osZOnQoc+fOBeDrX/86hx56KNXV1YwdO5Z//ud/3mG9s5dffhkzo33MVPvnX//615xxxhlUVVVxwAEH8LOf/azXsf/1r3/l+OOPJ5lMMnToUD70oQ+xcePbs8jefPNNzjnnHIYNG0ZlZSUHHHAAN910U0f9L3/5Sw477DCqqqpoaGjgmGOO4aWXXur9P6JID3j27d6V9vFPO7zSb4+HihpbITRio+s7ymIj6oiNa9jhFY6ofbt+VH2ufMwQLBz0f16ljwz6FrfUL5dCPIB/O6HUoUiBbfn0H0g/330qfF+LzxrNkBvP6JNrX3/99Vx55ZU8/fTTZLO5b/5mxo033siUKVNYtWoVl112Gf/yL//CAw88sNtrXXHFFXz1q1/l5ptv5n/+53+44IILOPbYY5k8eXKPYnnrrbc47bTTOO+88/jOd77Dhg0b+NjHPsa8efN46KGHAPjoRz9KGIY88sgj1NfXs3Tp0o7Ebvny5cybN4/rr7+es88+m5aWFp599tldtjCK7IuouY3mB1+k+sxZWCyk+eGF3ZaS6Nwtmn5lzW7HZ4mUyqBP3KwyRrS9+6wZkf7ohBNO4Oqrr96h7PLLL+94P2XKFG666SaOPfZYNm7cyLBhw3Z5rcsuu4xzzz0XgC9/+cvcfPPNPProoz1O3L75zW8yatQovve97xGL5R4ld9xxB0cffTTPPPMMc+bM4c033+QjH/kIhx2WGwPU+dorV64kiiLOP//8jm7hAw88sEf3FumtaHsr0ebcul+x8UNJzBxDNHHH3w9Lvr2Ad+U7p2PVex6KIFJsRUvczOx04CYgBL7n7td1qZ8I/BAYkj/mSne/L193FXAhkAU+6e67b0rohWBaPeGQQZ+/lqW+avUqpTlz5nQre/jhh/nqV7/Kyy+/zJYtW4jyi3W++eabu03cZs2a1fE+kUgwfPhw1q5d2+NYFi5cyLHHHtuRtLXHl0wmWbhwIXPmzOEzn/kMl156Kb/73e846aSTOPPMMznuuOMAOPLIIznxxBOZPn06c+fO5aSTTuLcc89l3LhxPY5BpKe8Lbd6QPsyGvHJI3Z7fHz/kX0ek8jeKEonvJmFwC3AGcCBwPvNrOtX6/8A7nL3w4F5wK35cw/Mfz4IOB24NX+9ggjqE4T711GO69lJ+amurt7h85IlSzjzzDOZPn06v/jFL1iwYAF33303kJu8sDtdJzaYWUfSVyj/9m//xuuvv86FF17I8uXLmTt3LhdddBEAsViMRx55hAcffJDDDz+cO++8k6lTp3Z0s4q0c3ea/vAC23/xdMer8XfPddS3PPbqDnXbf/E0Tb//e0d9858W0/KXVwFKsnyDSCEVa/TkHGCJuy9z9zbgTuA9XY5xoH0aTj2wKv/+PcCd7p5y99eBJfnrFUaYX4ohq8RNBp6nn36adDrNjTfeyLHHHsv06dNZs6Y44/oOOuggnnjiCTKZtwd0P/PMM7S2tnLwwQd3lI0fP56LLrqIn/70p9x6663cfvvtpFK5GXtmxtFHH81//Md/8PjjjzNnzhzuuOOOosQvA0hbluyarQTVFW9PDBgzpKM6HFbdfdJAp0kF4fBa4pOGkTh0AlabLMV/gUjBFOurxzjgrU6fVwBHdTnm88CDZvYJoBo4pdO5T3U5t3B9Ke0zf6IITbKVgWbatGlEUcQNN9zAP/3TP/Hcc8/xla98pSj3/tSnPsWtt97KRRddxGc/+1k2bNjAJZdcwimnnMKRRx4JwMc+9jHOOeccpk6dSktLC7/97W/Zf//9qaio4M9//jNPPPEEp5xyCqNHj+bll19m0aJFHbNlRTrEA6refRhBdQVBTfcJA4kDd/8noeKQ8X0VmUjR9ac24/cDd7j79WZ2DPBjMzt4Tye1M7OLgYsBJk7s+b6j7asyezZCW1jLQHPkkUfyjW98g+uvv57Pfe5zHHXUUdxwww2cddZZfX7v8ePH88ADD3DFFVfwjne8g6qqKs4880xuuOGGjmOy2Syf+MQnWLFiBVVVVRx77LHce++9ADQ0NPCXv/yFm266iS1btjBmzBguvPBCrrjiij6PXQonu6WZ1PzX8SgiHFZDcvaUnR7n7rQ+tZRoW275paCukuTRuTXSUgveIHlk7rzUC2+RWb3jxuJWEafqpBl9+F8hMnAUK3FbCUzo9Hl8vqyzC8mNYcPdnzSzJDC8h+fi7rcBt0Fun7+eBmZBQHZ1E65FeKUfOOmkk3Y53nJXXaCXXXYZl1122Q5lna8xY8aM3X5ut2JF982UOzv99NO7nXf88cfz+OOP7/Kc7373u7usO+yww7j//vt3e0/p/zIrN5NZsQniIdnVW6l4x+Sd7waSzpJ+eTWWbzHLrtpC8ojJYND20grCUXXEJw7r2DJqB1oTTaRDsRK3+cBUM5tCLumaB3ygyzHLgX8A7jCzmUASWA/cA/zMzL4BjAWmAs8UKjCLQpr/4ymG/J9T9nywiIjsIHHgWBJTR9H2yhpSC16HTBZ2tv9jYCSPOYBwZC3emia9ZB3geCqXpLVvNp48YjIcUbz4RQaaoiRu7p4xs0uBB8gt9XG7uy80s2uBBe5+D3A58F0zu4zcRIULPPf1fqGZ3QUsAjLAv7t7dud32guVuX8Cb8nsdjNZERHJyW7YTnrZeiC35VPFrIkEQ6qITRiKZ530G2uJNjftcI4lQipmTer4HBvbQNuStWRXbcnXxxGRPStappJfk+2+LmWf6/R+EXDcLs79EvClPgmsMqDyiiPIrttOUK/ZRiIie5J6aQWZ1zdAPCSorqBi1kTiE4YSn5Dbsi27egvp5Rt3OCeoSuyQuOWO20p6+UasMkE4tKpo8YsMZIO+icmSIbHpDUSNqVKHIiIyIHhrhnBELdVnztppfeU7p1PZg+tUnjCNyhOmFTY4kTI36Ed8tnePemu6xJGIiPRfUWMr6ddz3aNkox22hxKR4hn0LW4kY9CW+wYpIiI7l3ruTdJL1xGfMoLqfzxMu82IlIha3JLtLW5K3EREdiVqadth14GdLvkhIn1u0CduQWWC7BvbIFPYPRpFRMpJtLWFsK4nI9dEpC8pcatL0nztfMwLtm+9iEhZ8VQab0pBoFY2kVIb9ImbVeYG2HqLukplcLvgggs45RQtRC3dRa0ZCANiE4eVOhSRQW/QJ24kAqr+80iyzc2ljkQGsQsuuAAzw8yIx+MMHz6c448/nq997Ws0NTXt+QIifcTdCaoS1H3oOBLTRpc6HJFBb9AnblYVJ5xUl9umRaSETjjhBFavXs2bb77Jn/70Jz74wQ/yrW99iyOOOIK1a9eWOjwZpFqfXELTPX8rdRgikqfErSrfVarETUoskUgwevRoxo4dyyGHHMIll1zCk08+yfr167nyyit3OPbmm29mxowZJJNJpk6dype+9CUymVx3///7f/+P6dOnd7v+JZdcwvHHH9/jeNydr3/96+y3334kEgn2339/brzxxh2O+d3vfsfhhx9OVVUVQ4YMYc6cOfztb7k/8ul0ms985jOMHz+eiooKxowZw7x583r7zyIlFm1qImpsLXUYBddy7ytsv+4xtl/3GK0PLukob31oKZk3NpcwMpHdG/TruAVVCTydxTKDPoctS01/eKFbWXzycBIzx+KZLM0PLexef8AoElNHEbWmafnT4m71ieljiO83gqgxRctjr3Srrz7j0MIED4wbN44PfvCD/OhHP+L73/8+QRDw+c9/nh/84AfceOONzJo1i8WLF/Oxj32M1tZWvvCFL/DhD3+YL3/5yzz99NMcddRRAKRSKX7xi19w3XXX9fjet956K9dccw033XQTJ598Mn/84x/59Kc/TW1tLRdeeCFr1qzhvPPO44tf/CLnnXcera2t/O1vfyMWyz1Wbr75Zu666y5+8pOfsN9++7F27Voef/zxgv3bSHF4KjNgx7Z5S5otn7iPaPOOiaenMqR+/9oOZcn3TMczUUd58tyZhCOrqf/mGVh8x8lr2774KOm/r6X2iuNIzB7Xt/8R0qe2XvUwmVc37vnAvZA4ejy1n93pTp77ZNAnbiRCSEegBjfppw466CC2bdvGhg0bqKmp4Wtf+xq//vWvOf300wGYMmUKX/ziF/nkJz/JF77wBaZNm8ZRRx3Fj370o47E7d5776WlpYXzzz+/x/e97rrr+MQnPsHFF18MwNSpU3nllVf40pe+xIUXXsjq1atJp9Ocf/75TJ48GYCZM2d2nP/mm28ybdo0TjzxRMyMiRMncuSRRxboX0WKofmhl4i2txCOHVLqUHok/fc1bL7gt9ReezJbP30/2WW5ljMbkiQcX7fDsfEjxjDktrPw5jRbP/kHMks7tbIZtN73GrRmaPrOAsJxb5/rbVmidblxp62/XNTtunsrPms0Q757VkGu1c6qEwS1FbusjxpTeGNbQe/ZF5q+9xzN33m24NeNtqfwrbntLmMHjyz49cNJ9QW/Jihxw8zIvraVcGRhfvmkf9ld65fFwt3WB8n47utrKgraurYr7SvUmxkLFy6kpaWF973vfTssgJrNZmltbWX9+vWMGDGCD3/4w1xzzTXceOONxONxfvSjH3H22WczZEjP/gBv27aNFStW8M53vnOH8hNPPJGbbrqJ5uZmDj30UE477TQOPvhg5s6dy0knncS5557LhAkTAPjIRz7C3LlzOeCAA5g7dy5z587lrLPOIpFIFOhfpn8ws9OBm4AQ+J67X9elfiLwQ2BI/pgr3f2+ogfaQ1FziszKLcTGNxCOqMMqEySm9t9JCW1/W012xTYANp398x1+Js+ZQWzmcOq+8C4s3HWvysi/faxbmbdl2Hb1H7u11gFYRUhs5gjSz68pxH8Crb9/ldb/fZU1Y64vyPU6G/KdMwnG1HYrjzY2s+Ujvyv4/fpS1b8eXvBrWmWMui+8i6Bh4KxROOgTN4DUT16l8v87uNRhiOzUwoULqa+vZ9iwYSxbtgyAu+++m2nTum/OPXToUADmzZvHpz/9aX7/+99z3HHHcf/99/Pb3/62oHGFYcgf/vAH5s+fz8MPP8yvfvUrrrzySu6++27OPPNMZs2axeuvv85DDz3En/70Jz71qU9xzTXX8NRTT1FXVx5flMwsBG4B5gIrgPlmdo+7L+p02H8Ad7n7t83sQOA+YHLRg+2h7MZGWv/6KtVnzqJi1sRSh9NN810vkXlpHQDRllaabn5mh3qrSVD/33MJJw0hecbUvb6PJWLUf/20fYq1p6LtKVruWgjpwnX9pF/dSNMNT7Hl3/53t8dVf/xI4ocUvrWp0CpO2Y/YAQOzy77QlLiRW8vNW7TJvPQ/K1eu5Kc//SnnnnsuQRBw0EEHkUwmWbZsGe9+97t3eV5DQwNnnXUWP/7xj1m+fDlDhw7ltNN6/keorq6O8ePH85e//IUzzzyzo/zRRx9lypQpVFVVAblWwDlz5jBnzhyuvvpqTj/9dH7wgx90nFNTU8N73/te3vve93L11VczZswYHn30Uc46q7BdQiU0B1ji7ssAzOxO4D1A58TNgfZMtR5YVdQIe6l9+z+r6B9/HrZ/5TGInMQx49n+pcdIPfJ6rsLI/cvGAxq+dzaxg0ZCaMQPGtltTFp/F9RWUH3hEQW/bs3HjyTKdwXujNUmiE8bXvD7St/qH7+ZJZb44FSsury6b2TgaWtrY82aNURRxMaNG/nrX//KV77yFUaOHMlXvvIVIJcIXX311Vx99dWYGaeccgqZTIYXX3yRv/3tb3z1q1/tuN6HPvQhzjvvPBYvXswHP/hBwrB3f8yuuuoqLr/8cqZOncpJJ53EI488wre//W1uueUWAJ544gn++Mc/cuqppzJmzBhee+01XnjhBS688EIA/vu//5uxY8cya9Ysqqqq+PnPf04YhjttKRzAxgFvdfq8AjiqyzGfBx40s08A1cBOVzk2s4uBiwEmTixdS5e3tSdu8ZLFAODpLOsO/TaZlzfsUJ545ySG3HYW8elKOPZELVTlSYkbENQmIKZZpVJajz32GGPGjCEMQ+rr65k5cyaXXnop//7v/051dXXHcddccw1jxozhW9/6FpdffjmVlZVMmzaNCy64YIfrnXHGGdTX17N48WJ+/vOf9zqeSy65hKamJr785S/z8Y9/nAkTJnDdddd1JGb19fU8+eST3HLLLWzevJnRo0fzwQ9+kGuuuQbItdp94xvf4LXXXiOKImbOnMmvfvWrnS5VUubeD9zh7teb2THAj83sYHffYYNkd78NuA1g9uzZXoI4c3G0pnOtWYnStVqlX1rLlo/9L5mXNxBOqCN+1HjC0TUkz55Ocu7+JYtLpD+w9oHP5WT27Nm+YMGCHh+/5YsPQDxgyBVz+zAq6WuLFy/eYVajlJ/d/T82s2fdfXYx48knYp9399Pyn68CcPevdDpmIXC6u7+V/7wMONrd1+3qur19hhVSy5NLyLy+ntoPHNOn93F3Gr/+BNGaRgCSZ04jnFRP063zabz+SQCCUdWMXnn5bicWiJSD3jy/1OLWTs8FEem9+cBUM5sCrATmAR/ocsxy4B+AO8xsJpAE1hc1yl6omDWRxIwxfX6f9DMr2fZ/H8IqY3hLhsZvPNlRZ/UV1P3XyVRdeLiSNpEulLgBmEFoez5ORKQTd8+Y2aXAA+SW+rjd3Rea2bXAAne/B7gc+K6ZXUZuOP0F3o+7OoLKBFQWdsxvduU2os0tAGz+6L3Epg6l5ce5xbFHvfUZ0gtW0fi1x3F3kv84jdrLjy3o/UXKiRI3gOaIaNX2UkchIgNQfk22+7qUfa7T+0VA4ZdPLyBPZ0kv30g4tJr0krXExjYQG9dQkGu3PrSUjaf+eIey9FMrAKi+ZDbhsCrC0w4gedoBBbmfSLlT4gawLaL156/lVmMSERlk0svW0frEEqw2iW9vJdraUpDELf3Kho6kre5L7yI2bRjRphasKk4wrIqK0zTRQKS3ipa49WB18RuAk/Mfq4CR7j4kX5cFXszXLXf3swsaW01iQGz7IXvm7jvsKCDlox/3Lg543pJ7/lWdNIOme58nvv/eL8iaXbWNTfN+SeU/HUjTLfMBGPL9s6n+18KvUyYyGBUlcevJ6uLuflmn4z8BdN7bosXdZ/VZfKMS1Fx/PFFrG0FS67kNVGEYkk6ny25LJclpaWkhHi/t2mLlylMZiIdEjbntnSzZ+39ndyf9/BrWH/EdANoeWw5AxbumKGkTKaBiTdfpWF3c3duA9tXFd+X9QO8XntpLVhFiNXF8265XmJb+b8iQIaxdu5YoivZ8sAwY7k5zczMrV65k5Mj+vzXPQOSpDKSztL20EgCr2vXG5Luy9VN/6EjaOqu+pKgrtIiUvWJ1lfZkdXEAzGwSMAV4pFNx0swWABngOnfvtuniPq06nogBbUTbUoQju2/GKwPD8OHDWbFiBa+88kqpQ5ECi8fjjBo1qmz2OO1vwpG1WDKeG9cWDwnr97zhdvqltTRe/yQeORYazT94HoBhD/0LFgZseNcPsaGVJN+rtRVFCqk/Tk6YB/zS3TvvtjvJ3Vea2X7AI2b2orsv7XzSvqw6bhUxPN1GtF0tbgNZEAQl3SpIZKBKzBjb63Mav/4EzT97EdJvt3CPfPnSjq2oxmy9EquMax02kQIr1m/USmBCp8/j82U7M48u3aTuvjL/cxnwZ3Yc/7bPrDKXv2qCgogMNp6JyKzdiqfSvTqv9cGlVL7vQOpvOA2A5D8duMP+oUFdcsBt9i4yEBQrcetYXdzMEuSSs3u6HmRmM4AG4MlOZQ1mVpF/P5zcekiLup67L4KaJOnHV0NLppCXFRHp96LGVprve4HMys09P2dTM9HqRuKzx1L9yaMY8v2zqf/6qX0YpYi0K0ri5u4ZoH118cXAXe2ri5tZ56U95gF3dllVfCawwMz+DvyJ3Bi3giZu4bAaWr+/CG/ToHYRGVzaW9qsouczSZvuyI1nix80AgsCqv/1CGKThvRJfCKyo6KNcdvT6uL5z5/fyXlPAIf0ZWxWk1s+QmPcRGQwyazcTMtjrwK5sb57Em1P4S1pWu58CYCKEyf3ZXgishP9cXJCCUTUfOdkouaWUgciIlI0mbVb8ZY2EgeNIxhavftjl25i7cxvdUxGqL7saKxS6+qJFJsSN8BqK7B4AM0a4yYig0fiwHHEp4wgbNhz0rb5wt9BOiIYV0u0cjvxw0YXKUoR6UyJG7nEDcDblLiJyOARJOPQg10S1h/3faK1TYRThzLqxY+TeWUDsYO1GLJIKWiBHSCozI1x87bsHo4UESkf6RWbaHtt7S7rPZNl80W/I1rbBEDdNSdiFTHih47GAv35ECkFtbgB5BeI9IwSNxEZPNJL1hJtbCIxddRO6zee8VNSDy/D6ioY8cxHd1inTURKQ4kbYIGRfmYdQUXv9+cTERmosqu3YlWJndalX9lA6uFlBCOqGPnSxwlH1hQ5OhHZGbV152WfWk/0+vZShyEiUhTujremIbvz9Stbf59bJmTE4xcqaRPpR5S45Vl9BVGj1nETkUEik0vY4gd07yaNtray7fIHiR00gtjUYcWOTER2Q12leYlzJhKtby11GCIiReGZLFaT7NZV6u6sO/TbAFT9y2GlCE1EdkOJWzs3tT+KSNnzbETrM8uITxpO7XlHdqtv++tyssu3Ep8zjprPHluCCEVkd5SqtDODmJU6ChGRvpXJkn55NdnNTTut3vrZBwEY/uC/aMkPkX5Iv5XtAoNYwI7724uIlBfPT0awsPvjP/WXN0g/vZKKd00hqE8WOzQR6QElbnkWC7FkiLekSx2KiEjfyU9KILbj47/1D6+x4cQ7AKi77pQiByUiPaXELS+IxUk//Ba+VTNLRaR8eT5xsy6JW8tvFgOQPHcmiSPHFT0uEekZJW55QW01bfcvJ9qqmaUiUsaiKDc0JAx3KE49vIyK0/Zn6C/PL1FgItITStzyrDaBDUkocRORshYOr6Xuw8cTG98A5JZsbUYvAAAgAElEQVT/2PLvvyf7+haSZ07DTJO0RPozJW55Tpqab5xAtLml1KGIiPS59gSt6X8W0HTrfACqLphVypBEpAeUuOVZTW6fUt+mFjcRKV+tzyyj6Q8vvP3517mxbfU3nEZQo/2aRfo7LcCbZ7W5B1a0XZMTRKQ8Nf9xEZnlGwkaqgDILNtE6uFlVF0wi5pPH1Pi6ESkJ9TilhfU5VvcmttKHImISN/ILN8IQDiyDs9GrN3/mwBUXXh4KcMSkV5Q4pYX1OUWm/RmreMmIuWn8+LiiamjySxa3/G54vhJpQhJRPaCEre8oL6S1O+WEW1SV6mIlBd3p/Xx194uCAM2nPETAIb9/gMlikpE9kbREjczO93MXjGzJWZ25U7qbzCz5/OvV81sS6e6D5vZa/nXh/siviCZIPvsRny9ZpWKSJmJHG/LQDwkHF1P+vm1RCu3AxCbOaLEwYlIbxRlcoKZhcAtwFxgBTDfzO5x90Xtx7j7ZZ2O/wRweP79UOA/gdmAA8/mz91cyBjdnWBiLZG2vBKRMmNhQNW7DgQgu2obWz7xBwCqPzGHcPKQUoYmIr1UrBa3OcASd1/m7m3AncB7dnP8+4Gf59+fBjzk7pvyydpDwOl9EWTFhdMJxmg6vIiUl6ixlagpNwxkzbhv5JYAScaov+kMLbgrMsAUK3EbB7zV6fOKfFk3ZjYJmAI80ptzzexiM1tgZgvWr1/ftXqPzAwyjmejXp8rItKftb2yhsa7niGzsmMECnWfP0lJm8gA1B8nJ8wDfunu2d6c5O63uftsd589YsRejtmIHFyJm4iUF2/LYBUxsq9sAiA2Yzg1nz22xFGJyN4oVuK2EpjQ6fP4fNnOzOPtbtLenrtv3PA9HyUiMqB4Ko1VxMm8lkvchj3wz1jQH7+3i8ieFOs3dz4w1cymmFmCXHJ2T9eDzGwG0AA82an4AeBUM2swswbg1HxZwVlgWMzwdK8a+0RE+jVP5VrcMq9thGSMcHxdqUMSkb1UlMTN3TPApeQSrsXAXe6+0MyuNbOzOx06D7jTO60U6e6bgC+QS/7mA9fmywrOPE7bQ29po3kRKSsdiduyzcSmDFFrm8gAVrS9St39PuC+LmWf6/L587s493bg9j4LLi+sryH70iaiTS2EI2v6+nYiIn0uamyl4qBxWFWCxuWPE07S8h8iA5m+dnVi9QmC/eqINqnFTUR6Zl8WF+9rnkrTePd8Whe8TjishvSzqwknqJtUZCArWovbQBBlU1RdcQTRRiVuIrJn+7K4eDF4KpP72dzG9q8/DkBmYe+XSxKR/kMtbp0EtRVYPCS7sbnUoYjIwLAvi4v3uaCukviMMQBElkvi6q47pVi3F5E+oMStk6A+CYBvUYubiPTIviwu3ufcHdrnerVmCEbXUHHCpGLdXkT6gLpKO7G6fOK2rbXEkYhIGdrt4uJmdjFwMcDEiRMLcsPtd/y1471vayMYVlmQ64pI6ajFrZMgmQAgamorcSQiMkDsy+LiOyjI7i+dpF54a4fPbfNXEQyr2ufrikhpKXHrJBhaTerXy4hWN5U6FBEZGPZlcfE+FTW+3XMQ1FVCJiIYrsRNZKBT4tZJUJnA17cRrdXkBBHZs31ZXLzPZXP7LiePPQA8ILtoE9UfPaJotxeRvqExbp145IQzGoje2l7qUERkgNiXxcX7kmcigvpKEtPHkPrVa/j2NIljJuz5RBHp19TitgMnfuIobLwG8IrIwJZ5ayOEuUd8dl0TVlfRMXNeRAYutbh1YkGAZ4vXkyEi0lcqZk3CW9MAZFdsIxiqL6Qi5UCJWxfmYKHhmSwWC0sdjojIXqk49O1u0dZfLSZ2yMgSRiMihaKu0q4sgOoY0Wat5SYiA1PUmCKzeguejfCWXKtbYvbYEkclIoWgxK2rWIBVxom07ZWIDFCZ5Rtpvv9Fsmu3kV2dm2yVeKd2TBApB0rcuoiNbCB156tE67SWm4gMTO2rjoTDamhbsAqA+Mx9X9RXREpPiVsX8fENRG81KnETkYErv4YboZFdvhWA2IzhJQxIRApFiVtXlSGxY0aTVeImIgNVlJ8dHwT41hQYWG2itDGJSEEocesi29hM5UcPIlqvxE1EBiaPolyyFhjR1lastgIL9LgXKQdaDqSLoCq/0fzmlhJHIiKydxIzxhCfOAwA35YiqK8ocUQiUihK3LqwZByA7DYtByIiA1NQVQFVuWQt2prCtGOCSNlQ23kXVpFL3Ly5rcSRiIjsncyaLbQtXQdAtLVVLW4iZaRoiZuZnW5mr5jZEjO7chfHnG9mi8xsoZn9rFN51syez7/u6dM48y1utGX68jYiIn0mvWQdqQWvA+BqcRMpK0XpKjWzELgFmAusAOab2T3uvqjTMVOBq4Dj3H2zmXXen6XF3WcVI9agNkm0qJH002uKcTsRkcKLHPKTEaItrcSmDytxQCJSKMVqcZsDLHH3Ze7eBtwJvKfLMR8FbnH3zQDuvq5Ise3AwoCgppJoVSOeyZYiBBGRfeJtGSwR4pks2eVbCScNKXVIIlIgxUrcxgFvdfq8Il/W2TRgmpk9bmZPmdnpneqSZrYgX37Ozm5gZhfnj1mwfv36fQrWRicJDx5KtEHbXonIwOOpDFYRJ7u6ETIRsSlK3ETKRX+anBADpgInAe8Hvmtm7U+bSe4+G/gAcKOZ7d/1ZHe/zd1nu/vsESP2bWsXT2aJHz9WuyeIyIDkqTRWEev48hmMqC5xRCJSKMVK3FYCEzp9Hp8v62wFcI+7p939deBVcokc7r4y/3MZ8Gfg8L4M1hIxrCZOdr1a3ERk4Kk67RCSc/Yj2pRbjzIYWlniiESkUIqVuM0HpprZFDNLAPOArrNDf0uutQ0zG06u63SZmTWYWUWn8uOARfQhq0xgtXG1uInIgBRUVxBUVxBtzLe4DVPiJlIuijKr1N0zZnYp8AAQAre7+0IzuxZY4O735OtONbNFQBb4rLtvNLNjge+YWUQu0byu82zUvhDUJbEaJW4iMvB4OkvbopXEJgzF21vchlWVOCoRKZSi7Zzg7vcB93Up+1yn9w58Jv/qfMwTwCHFiLFdUFuB1STIvt5YzNuKiOwzT6VJPfcmVpkg2qiuUpFy058mJ/QbiUMm0HLD34lWby91KCIiveKZCACLBWQ3NGPVcaxCuxuKlAslbjsRJOMEdZW5qfQiIgNJNpe4EQZkl24inNJQ2nhEpKCUuO1EtL2V+MljiRq10bxIubOcj5rZI2b2Qr7snWZ2fqlj2xtvt7iFZF7dSGyadk0QKSdK3HbCW9OE0+vwwEsdioj0vWuBC4HbgIn5shXAFSWLaF9kczu+ZNc1kXllI7H91OImUk6UuO2EVeY2mjfLrUAuImXtAuBMd78TaP+29jqwX8ki2gfh6Hpq3n8U0ZrcUiDxw0eXOCIRKSQlbjthlYncz/oEWU1QECl3IdA+oLU9cavpVDagWBAQJBPQmmt5ix80ssQRiUghKXHbCQsDMMslbquUuImUuT8A3+i00LcBXwDuLWlUeymzZiutz75B1NyWK6jUjFKRctLjxM3MTjazKfn3Y8zsh2b2AzMry3Z4q4hjdQkizSwVKXeXAWOArUA9uZa2SQzQMW7ZNVtpe+EtvDU3zKN96IeIlIfetLjdSm5HA4DrgTgQkRvQW3YqT5xB67dfUoubSBnLt64NB84jNzHhaGB/d3+vuw/IX35PpSEeQnN74qYWN5Fy0pvf6HHuvtzMYsBp5L6RtgGr+iSyEgtH10Is0Bg3kTLm7m5mLwK17r4OWFfqmPaVpzJYRQxvyXWVWpVa3ETKSW9a3LaZ2SjgRGCRu7f3IZblUyGzagvJiw8iUoubSLn7GzCt1EEUiremsYo43pQGA0uqxU2knPTmN/pmYD6QAD6dLzsOeLnQQfUH0eZm4rNH0nb3G6UORUT61p+B+83sDuAt3p5ZirvfXqKY9oq7k93aTDiijszWNVhtBRZoDppIOelx4ubuXzWz3wBZd1+aL14JXNQnkZWYVeWWBIm2p0ociYj0sePIrdt2YpdyBwZU4mZm1Lz3HUTbW0n98GWC+opShyQiBdarNnR3f7X9vZmdDETu/mjBo+oHgvxMrKhJiZtIOXP3k0sdQyFZLCRsqCbalsLqk6UOR0QKrDfLgTxqZsfl318B3An8zMyu7qvgSqljEd6YEW3TnqUi5czMGszsQ2Z2Vf7ngNwnKmpto/E3z5J+Yz2+tZWgTi1uIuWmN4MfDgaeyr//KHAyuanzHyt0UP1Be+JGRUh2+dbSBiMifcbMjgGWknuWHQr8G7A0Xz6wpCOiLc14OptvcVPiJlJuepO4BYCb2f6Aufsid38LGJDfTPfEKmIkpk8i89fVStxEytuNwMfd/Vh3f7+7HwdcAnyzxHH1mkcRkNv2yrem1OImUoZ6M8btr8C3yK0w/huAfBK3oQ/iKjkzIz5pCIASN5HyNg24q0vZL4H/KUEs+yabnxAbmsa4iZSp3rS4XQBsAV4APp8vmwHcVNiQ+o/0us0kzjuAjBI3kXL2GjCvS9l55LpPB5Z8ixtBoDFuImWqN8uBbASu7lL2+4JH1I9Em5uIzx5J9gUlbiJl7NPA/5rZJ4E3gcnAVODMUga1V2Ih4dghWDzEWzIa4yZShnozqzRuZv9lZsvMrDX/87/MLNGXAZZSUF2BDalQV6lIGXP3J4D9yQ0FeZbcYuMH5MsHlHBIFdWnHYIlco9ltbiJlJ/edJV+DTiF3Myrw/I/3wV8tQ/i6hesugKLB2TXNO75YBEZkMxsHIC7/8Tdv+buPyE3EWtsD88/3cxeMbMlZnblLo4538wWmdlCM/tZAcPfKd+aW8JIY9xEyk9vErfzgLPd/UF3f8XdHwTeC5zfk5P35eFmZh82s9fyrw/3IuZ9ElTnvq16axueyRbrtiJSXL8FxncpG09+EtbumFkI3AKcARwIvN/MDuxyzFTgKuA4dz+It7cMLLjMys1sv/sZMmu3AWjnBJEy1JtZpdbL8rcPePvhNhdYAcw3s3vcfVGnYzo/3Dab2ch8+VDgP4HZ5LageTZ/7uZexL5XgpokuEE8ILu6kdiE+r6+pYgU3zR3f7Fzgbu/aGYzenDuHGCJuy8DMLM7gfcAizod81HglvZnlruvK0zY3XlbBm9M4Y1tgLpKRcpRb1rc7gbuNbPTzGymmZ1O7pvq3T04t+Ph5u5t5HZdeE+XY3b1cDsNeMjdN+XrHgJO70Xcey0cUUti7CiiZds0zk2kfK03swM6F+Q/b+zBuePIbUzfbkW+rLNpwDQze9zMnso/O7sxs4vNbIGZLVi/fn0vwn+bR7nlQLwpnbumukpFyk5vErf/CzxMruWsfQDvn4DP9uDcfXm49eTcgjz0diacmGtlU+ImUrZuB35lZmea2YFmdhbwK+B7Bbp+jNws1ZOA9wPfNbMhXQ9y99vcfba7zx4xYsTe3SmbWw7Et6vFTaRc7bar1Mze1aXoz/mXkeu2BDgeeKRAsbQ/3MYDfzGzQ3p6srvfBtwGMHv2bN/D4T2WXr2BxHv3I/vmlkJdUkT6l+uANPB1YAKwnFzSdkMPzl2ZP6fd+HxZZyuAp909DbxuZq+Se9bN38e4u8uv4xZtyyVuWg5EpPzsaYzb93dR3p4YtSdw++3hOvvycFtJLpnrfO6f93C/gvGWNmLTG8gs6/MhdSJSGicCv3T3/zazMeRmyh8MjATW7OHc+cBUM5tC7lk1D/hAl2N+S66l7QdmNpxc78KyAsbfwWqSxCYOIzM/18urFjeR8rPbrlJ3n7KL13751xR331PSBp0ebvl13+YB93Q55rfkE7QuD7cHgFPNrMHMGoBT82VFYVUJbEQl2aVK3ETK1K1A+7Tx68l9oY3It+DvjrtngEvJPZMWA3e5+0Izu9bMzs4f9gCw0cwWkR9ekl/QvODi44dS9Q8H4ltaIRFiyXhf3EZESqg3s0r3mrtnzKz94RYCt7c/3IAF7n4Pbydoi8g9RDsebmb2Bd7uVrjW3TcVI27IL8JbmyCzpGi3FJHiGufuy80sRm7i00SgDVjVk5Pd/T7gvi5ln+v03oHP5F9F4dtSWgpEpEwVJXGDfXu4ufvt5AYQF51VV2AxI7upCU9lsIqi/ZOJSHFsM7NR5LpHF7p7Y75nYMA1V6WeX07b4lVEW1OYuklFypKykD0IG6oxC7FESOb1zcRn7OVsLxHpr24m16Kf4O3FcY8DXi5ZRHvJ09nca1uKQEuBiJQlJW57EBvXQGL6BHxTiuxSJW4i5cbdv2pmvwGy7r40X7wSuKiEYe2dbAShEW1tVYubSJlS4tYDsQOGAmicm0iZcvdXd/d5oPAowoIA35YinNxtqTgRKQO9WYB30Gp55jUqPjSDzFIlbiLSj0UOYaAxbiJlTC1uPZF1wil1ZJ5R4iYi/Vc4sg5LxIi2tmqMm0iZUuLWA0FtkmBYJW1ay01E+rHEtNG4u5YDESlj6irtgaAmCVUhmTc24/m9AEVE+iNvTkPW1VUqUqbU4tYDQW0SCwyrjpN9cwux/YaWOiQRkW6aHnwJb85vMK+uUpGypBa3HgiH1RDUVYNB5uUNpQ5HRGTnshFkcr0CanETKU9K3HogHFFL1ckz8E0p0ouVuIlIP5WNcjNLQWPcRMqUErcesoZKgnE1ZBatL3UoIiI75ZHj2VziphY3kfKkxK2Hmn7zLMkLDiSzWImbiPRT2Qgy+RY3JW4iZUmJWw9ZVYJgVCXpxRtw91KHIyLSTXy/EZDNPdatJlHiaESkLyhx66GgvgqrjuFbWonWNpY6HBGRbioOm4i15N5btRI3kXKkxK2HgrpKMLDauMa5iUi/5JksUVMKAKuKlzgaEekLStx6KKyvBCAYXa2ZpSLSL23/xTOQyAJK3ETKlRK3HgqG1pA4dDyedU1QEJH+KYrwtgiSMSzU412kHOk3u4eCqgTJd0whHFFDeqESNxHph7IRtGXV2iZSxpS49YK3ZYjPGUP6hbWaWSoi/YpHDg7eliWoVuImUq6UuPVCy19fJTxsCL6pheyKbaUOR0TkbVFuqytvzWpGqUgZU+LWC0F9FQQOoZH++5pShyMi8jYzEodOIFrTrK5SkTJWtMTNzE43s1fMbImZXbmT+gvMbL2ZPZ9/XdSpLtup/J5ixdxVUJebWWrDk6SfV+ImIv2HhQHJd0wmWt6IqatUpGzFinETMwuBW4C5wApgvpnd4+6Luhz6C3e/dCeXaHH3WX0d554EQ3KJW/yIUaT/vrbE0YiIvM0jx1NpvCVDMCRZ6nBEpI8Uq8VtDrDE3Ze5extwJ/CeIt27YMIh1QDEDhmhrlIR6Ve8sZXGO5/GxlQocRMpY8VK3MYBb3X6vCJf1tX7zOwFM/ulmU3oVJ40swVm9pSZndOnke6GxUOSx08jHFJDdskmosZUqUIREdmBt09O2JIiGFZZ4mhEpK/0p8kJ9wKT3f1Q4CHgh53qJrn7bOADwI1mtn/Xk83s4nxyt2D9+r5bZy0xdRTxg0aBQ/rFdX12HxGRXsnklijyLa0EQ5W4iZSrYiVuK4HOLWjj82Ud3H2ju7c3YX0PeEenupX5n8uAPwOHd72Bu9/m7rPdffaIESMKG30nUUsbNrYKYgHpZ1f12X1ERHrDs7mtrrwtIhhWVeJoRKSvFCtxmw9MNbMpZpYA5gE7zA41szGdPp4NLM6XN5hZRf79cOA4oOukhqLJrtlK24tvEjtkOG3zlbiJSD+RyXWV0pZVV6lIGSvKrFJ3z5jZpcADQAjc7u4LzexaYIG73wN80szOBjLAJuCC/Okzge+YWUQu0bxuJ7NRiyZoyE1QiB87jvTDy0sVhojIDoK6JLERDUTrW9RVKlLGipK4Abj7fcB9Xco+1+n9VcBVOznvCeCQPg+wh4K6SgiN2PQGWm75G9GWlo5lQkRESiWorSQIEviWNnWVipSx/jQ5YUCwwAjqq7Bhuen2bQvUXSoipeepDNkNTRCYWtxEypgSt70QNlSD5caTpJ9ZuYejRUT6XnrpOrLZRqw6pjFuImWsaF2l5aRi1kQqDp9E87T5tClxE5F+wNvSuZ8tGaxeC/CKlCu1uO2FoK6SoDZJfM442p5eibuXOiQRGeQ8lcGzjtVUYKEe7SLlSr/de6nt5VXEjx9LtKaR7LLNpQ5HRAY5T2WgLdL4NpEyp8RtL7UtXo2NSACQeuzNEkcjIoOdZ7J4W5agQd2kIuVMidteCofV4Kk2bGglbX9R4iYipZWYPobM/A0EDWpxEylnStz2UjC8Bm9NUzF3MiklbiJSYrFxDWSfXacWN5Eyp8RtL4XDagGIHzee7NLNZFdtK3FEIjKYZTc34eaYxriJlDUlbnspHFYNgRGb2gBA6jFtfyUipdPy6MvE545XV6lImVPitpcsFlL7gWOoPGU6VpPQODcRKSlPZyGlyQki5U6J2z6weIjFQhLHTSD1p9dLHY6IlICZnW5mr5jZEjO7cif1F5jZejN7Pv+6qE8CSUd4OlKLm0iZU+K2D7Jbmml64EUSp00hs3gD2RVbSx2SiBSRmYXALcAZwIHA+83swJ0c+gt3n5V/fa8vYvFMFtKRtrsSKXNK3PaBxUOyq7YQP2Q4AK0PLi1xRCJSZHOAJe6+zN3bgDuB95QkkmwE6Yhwv4aS3F5EikOJ2z4Iqiuwmgo8iAjG1pJ6QImbyCAzDnir0+cV+bKu3mdmL5jZL81sws4uZGYXm9kCM1uwfv36XgdizSHpJ9cQm1jf63NFZOBQ4raPwpF1ZNdtJ3nq/rQ+tBTPRqUOSUT6l3uBye5+KPAQ8MOdHeTut7n7bHefPWLEiF7fxDe0ES3frg3mRcqcErd9FBtVh7e0kThlMr65lfSCVaUOSUSKZyXQuQVtfL6sg7tvdPdU/uP3gHcUOgiPnCjTRjChVhvMi5Q5/Ybvo3BUPeHoehLHjAOD1geWlDokESme+cBUM5tiZglgHnBP5wPMbEynj2cDiwseRTpDsF8l8WNGF/zSItK/KHHbR2FDNdVnHEp8vxHEjxxH6+9fK3VIIlIk7p4BLgUeIJeQ3eXuC83sWjM7O3/YJ81soZn9HfgkcEHB48h67k0iVuhLi0g/o9/yAvF0huQ5M9h+9R/JrtxGOK6u1CGJSBG4+33AfV3KPtfp/VXAVX0aRJQbW2uJsE9vIyKlpxa3Aki/uYHtP32SitOmANDyu5dLHJGIDCbtk6Isqe/iIuVOiVsBhENrwMGSRmz6MFp/o8RNRIoo31VqyXiJAxGRvla0xG1ftoUxsw+b2Wv514eLFXNPBbVJrDZJZtUWkufMIPXnN4g2t5Q6LBEZJIKaClq+uwhDXaUi5a4oidu+bAtjZkOB/wSOIrdK+X+aWb9bGjw2ZgiZNVtJnjMDMhGt//tqqUMSkcEiFpB5ajVBTUWpIxGRPlasFrd92RbmNOAhd9/k7pvJLWB5eh/FuddiY4dAOkswuZZwfB3Nv3ip1CGJyCCRXd9E7LDhWIMSN5FyV6zEbV+2henRufu6Xcy+io1roGL2FMK6SirffzCpB5aSXd9U9DhEZPDJrtxC5ScPw2o0OUGk3PWnyQk92hZmV/Z1u5h9ZYkYFYeMJ6iqoOqfD4VMRMtdC4seh4gMPt6SBsAqEyWORET6WrESt33ZFmaP5/YXns6SXraOcL8GYgePpOWnL5Q6JBEZBKLW9sRNLW4i5a5Yidu+bAvzAHCqmTXkJyWcmi/rd7w1Tcujr5B+Yz1V/3wobU+uILN0U6nDEpEy581tAFiVWtxEyl1RErd92RbG3TcBXyCX/M0Hrs2X9TtBbZKgoYrMW5uo/MAhYNB8x/OlDktEypynMng6UuImMggUbYybu9/n7tPcfX93/1K+7HPufk/+/VXufpC7H+buJ7v7y53Ovd3dD8i/flCsmPdGbMIwsmu3EoyoouLdU2n63nN4OlvqsESkjAWxClr++zmCaiVuIuWuP01OKAvxycPBIfPGBqo/NptoTSOt975S6rBEpJylIrJLtmJV2jlBpNwpcSuwYGg1QX0l2fXbSJ4xlXBCHU3/s6DUYYlIGYuaWwgPG6bJCSKDgBK3AjMzqt59GMnjp2FhQNVH30HqoWVklmwsdWgiUqaibBsV5+yvFjeRQUCJWx8IknHMDHen+sLDIRbQ+K1nSh2WiJSrbATprBI3kUFAiVsfST3/Js0PvkQ4to7KDxxC83efI7uxudRhiUg5csfbIqxSiZtIudOAiL4SC8mu2kJ2SxO1nz2Wlh/9naZb51N3zYmljkxEyoy755YDCfVdvKe2bdvGunXrSKfTpQ5Fylw8HmfkyJHU1dUV5HpK3PpI/ICRpJ59g/Sra0nO2Y+Kf5xK0zefpvb/HKtvxSJSWAHQqmWHemrbtm2sXbuWcePGUVlZiZmVOiQpU+5OS0sLK1fmNnwqRPKmr2d9JEgmiE0cRnrJWjwbUft/jyPa0EzT958rdWgiUmaihdtJ/2VVqcMYMNatW8e4ceOoqqpS0iZ9ysyoqqpi3LhxrFu3riDXVOLWhxLTRuOpDJnlG0mcMInECRPZ/uXHiPLb04iIFIJvSkGblzqMASOdTlNZWVnqMGQQqaysLFi3vBK3PhSOHULisAkEQ6sxM+q+/A9EqxtpulkzTEWkMLwt8/+3d+fxUVV348c/31mzEshKAoGwbxLZ3KGgghILaGmtUCygRcGN0toHER984HEDXpZFsK2AWKiAFuVRXOpCXfFnEZBVAsoqIEvYAiGTZWbO74+5GRLCTpjJkO/79Royc865937nZjj5zr33nIvUdWNLjw13KBFFj7SpUKrKz5smbpeQiBDVIQt7QgwA7s4Ncd/WjGMTluI/4glzdEqpy4G/oBh7mwRs9ePCHYpSKgQ0cQsB776jlGwMXH9S65mbMIeLODbhqzBHpZS6HJgS6/SLnilVVWDjxo2ICCtW6B1/qo4rMvIAACAASURBVCtN3EKgdOt+ipZtxe8pwdUuneiBV1Iw6Wu8P+jdFJRSF8kXyNjEaQ9zICoUROSMj6ysrItaf7NmzdizZw/t2rWrmoBVldPELQRcrTPAbyjJDRx1S5jQHYlycGT4vzBGvyYrpS6c8VuJm0sTt5pgz549wcebb74JwLfffhssW758+SmXKyk5t0FxdrudunXr4nBcnrOFnet+qM40cQsBe0IMjgZJlOT+hCn2Yq8bT61x3Sj+YDNFb28Md3hKqUhmJW64Ls8/tKqiunXrBh+JiYkApKSkBMtSUlKC7caNG8f9999PYmIiPXr0AOD5558nOzub2NhYMjIyuPvuuytMU3HyqdKy14sWLSInJ4eYmBiaNm3K/PnzzxhnXl4e/fv3JzMzk+joaFq2bMm0adMqtXv11Vdp164dUVFRJCcn06tXLwoKCoDAHGhTpkyhZcuWuN1u0tLS+M1vflNhXzz//PMV1nf33XfTs2fP4Otrr72WBx54gFGjRlG3bl2aN28OwJw5c7jqqquoVasWKSkp9OnThy1btlRY1549exg4cCCpqalERUXRsmVLXn31VXw+H/Xr12fSpEkV2h85coSYmBgWLlx4xn1zsTRxCxF3+wZQ4qN4Q2ASvtiHrsbRNpUjD72P/7AOVFBKXRhHvdoUTliFGB0lqSr685//TFZWFsuWLeOll14CAqdap0yZwvr161m4cCHff/89v/3tb8+6rscee4z77ruPtWvXcscddzB48GC2b99+2vYej4cOHTqwePFiNmzYwKhRoxg5ciQLFiwItvnrX//KvffeS79+/Vi1ahWffPIJN910Ez5fYDLpUaNG8eSTTzJixAjWr1/Pe++9R9u2bc97P7z66qsUFhby6aef8u677wKBI2/jxo1j1apVfPDBB5SWltKnTx+8Xi8ABQUFdOnShY0bN/Laa6+xYcMGJk+ejNvtxm6387vf/Y5Zs2ZV2M68efOIj4/njjvuOO8Yz4d+RQsRe2IczmZpiPWtWJx26rxyB3nXzOTI8H+R+I++YY5QKRWJxGHHv+c4EqXd+cU4MuJflK7eG/LtOtvVpfaUnEuy7i5dujB69OgKZY8++mjweaNGjZg6dSrXX389Bw8eJCkp6bTr+sMf/kDfvoG/U88++yzTpk3j888/P+01dQ0aNOC//uu/Kmzr66+/Zv78+fTv3x9jDGPHjmX48OGMGjUq2C47OxuAw4cPM3nyZCZNmsSwYcOC9Z06dTr3HWBp2LAhU6dOrTAlx3333VehzezZs8nIyGDNmjV07NiROXPmsHfvXr766ivS0tIAaNy4cbD9kCFDeOaZZ1i6dCmdO3cGYNasWQwePBin89LeHUmPuIVQdOfmuNvUC752dcwg/omf4Xl1LZ7/yw1jZEqpSOU7VIDzxgwkTm+lpyq6+uqrK5UtWbKEHj16kJmZSXx8PN27dwdgx44dZ1xX+cEKLpeL5ORk9u3bd9r2Xq+Xp59+muzsbJKSkoiLi+OVV14Jbmfnzp3s37+fW2655ZTLr1u3jtLS0tPWn4+rrrqq0jxqK1eu5PbbbycrK4v4+HiaNWsGnNgPK1euJDs7O5i0nSwzM5OcnBxmzpwJwIoVK1izZg1Dhgy56HjPRr+ihZgxBu+2A9hTa2GLcxP/RBeK3tnE4SGLcXZIx9GwdrhDVEpFEN++o7hvbwx7/OEOJaJdqqNe4RQbW3FS5s2bN9OrVy+GDBnCuHHjSEpKYsuWLfz85z8/60X7LperwmsRwe8//WfuueeeY/LkyUyePJns7Gzi4+MZP348S5cuvfA3dBKbzVZpgN+p7k5w8n7Iz8+nR48e9OjRgzlz5pCWlkZJSQlXXnnleQ1eGDZsGHfeeSdTp05l1qxZdOvWLZgAXkp6xC3ETGEJnqWbKFq5DQBxOUh8/U7w+jn0q39iir1hjlApFUn8R4sBsCXqLZzUmS1btozS0lKmTJnC9ddfT4sWLdi799KcHv7iiy/o3bs3gwYNon379jRt2pTvv/8+WJ+ZmUlqaiofffTRKZdv27YtTqfztPUAqamp/PTTiXv0GmNYvXr1WWNbv349hw8fZvz48XTt2pWWLVty4MCBCm06duzI2rVrz3hUMScnh5SUFGbMmMGCBQsqnX69VDRxCzFbrBtXm/p4t+bh3X8UAEezJOrMuYPSFT+R/4cPwhyhUiqS+I8UAWBL0zsnqDNr3rw5fr+fyZMns23bNt58802ee+65S7KtFi1asGTJEr788ks2bdrEyJEjWbNmTbBeRBgzZgwvvPAC48ePZ+PGjaxfv56pU6eSn59PnTp1GD58OKNHj+all17ihx9+YPXq1UyYMCG4ju7du/Pqq6/yySefsHHjRh5++OFzSkQbNWqE0+nkhRdeYOvWrXz00UcVrscDgqNJe/fuzSeffMK2bdv4+OOPeeONN4JtbDYbQ4YMYcyYMTidzuA1gJdayBI3EekpIptEZLOIjDpDu1+KiBGRTtbrLBHxiMhq6/G3UMV8qbizM5EYF0Vfbw7OwRR9RyviRt7A8b+uoOCF/4Q5QqVUpPDnBxI3e129V6k6s6uuuopJkyYxdepUWrduzbRp05g8efIl2da4ceO45ppruO2227jhhhsoKSmpMMgA4OGHH2bGjBnMmzeP7OxsunXrxpIlS7DbA3MSTpw4kSeffJLnn3+eNm3a0LNnT9atWxdc/r//+7/p3r07ffv2pVu3bmRkZNC7d++zxpaRkcGcOXNYvHgxrVu3ZvTo0ZX2Q3x8PF9++SVNmzblzjvvpFWrVgwfPpzi4uIK7YYMGYLX62XQoEG43e4L3V3nRUIxAayI2IHvgR7ALmA50N8Ys+GkdvHAe4ALeNgYs0JEsoB3jTFXnOv2OnXqZKr77TpKtx/A82ku7o5ZuLMzATC+wOnSorc3kvjGr4nu2zrMUSoVOURkpTHm/IecVUPn04ft7zmbqLuaEd2tDc5GiZc4sstDbm4urVq1CncY6jLw7bff0rFjR3Jzc2nZsuUZ257pc3c+/VeojrhdDWw2xmw1xpQArwG3n6LdU8AEoChEcYWNMysZZ8t0bLVjgmVit5E4/5c4r6nPoQGLKP5sWxgjVEpFgtIlOzn20OfYM+LDHYpSNUZRURG7du3i8ccfJycn56xJW1UKVeJWD9hZ7vUuqyxIRDoAmcaY906xfCMRWSUin4tIl0sYZ0hFX9cUZ4OK8+ZItJOkd/rjaFSbgz+fT/GnmrwppU7NGAM+Q0y/K7C5dToQpULl73//Ow0bNmTv3r1Mnz49pNuuFoMTRMQGTAIePUX1HqCBMaY98EdgvojUOsU67heRFSKyIi8v79IGXIWMMRSv3UnRihMJmj05luRPB2PPqs3Bn8+jaMmWM6xBKVVjlfqwX5GIvZ2eIlUqlIYNG4bP52PNmjUVJuYNhVAlbruBzHKv61tlZeKBK4DPRGQ7cC2wWEQ6GWOKjTEHAYwxK4EtQPOTN2CMmWGM6WSM6VR2r7ZIICL4C4ooWbeL0p2HguX2tDiSPx2EvWkiB3PmcXzO2Yc4K6VqFuPxYm9RB+L0dldK1RShStyWA81EpJGIuIB+wOKySmNMvjEm2RiTZYzJAv4D9LEGJ6RYgxsQkcZAM2BriOIOiairm2BLjMXzxSZ8+YXBcntqHClf3IO7WxZHBr/F0TGfYM4w4aFSqmYxhaWITZM2pWqSkCRuxhgv8DDwIZAL/NMY852I/K+I9DnL4j8D1orIauANYJgx5tBZloko4rARc1NrxCYUfvwd/qITMz/bakeT9P4AYu5tz7Gnv+Bgr/n48o6HMVqlVHVhPKXgENDcTakaI2S3vDLGvA+8f1LZk6dp263c8zeBNy9pcNWALT6K6JtaU/jROnz78rE1TA7WidNO7Vl9cHZMJ/8PH7K/3d9InNcXd7dGYYxYKRVuhf9YC3YbmrkpVXNUi8EJKsCRVou4X12Fs1zSVkZEiHvwalL+MwSJdXLgxjkcHvZOcPJNpVR4XOjk4lXB3b0xzg7piFtvO61UTaGJWzVjiw7cyLd05yEK/70B4614TZurfTqpq4YR9+h1FM78ln2tplM4f61e+6ZUGFjX374I5ACtgf4iUmnmbGty8d8Dy6py++4bGlBrRFfi+l1TlatVSlVjmrhVU8ZTgvfHgxR+tA5TXFqhzhbrIuH5W0lZNgR73TgOD1hE3jWzdM43pUIv7JOLi4gOUFCqBtHErZpyNa9LdNeW+PKOcfy9NfiPVe7vXZ3qkbL8furMuQP/vgIO3DSHvJ/NxvPuJj0Cp1RoXOzk4hetZNMeilfvuBSrVkpVQ5q4VWPOxinE3NoWv6eU4++swl9QOXkTu42Yge1I2/QICVN64tt+hEO9F7A/+68UvPgN/iOeMESulIKzTi5+ctsLmkTcu/swpdsOXESUKtIMHjw4cKRVBKfTSXJyMp07d2bixIkcP66zDlzuNHGr5hx1E4jr3Q5nqwwk1g1Yt7k5iUQ7ifv9taRt+T115v4CcdrJf/h99qT/mUMDF1H04WZMiTfU4St1ubvgycVPXtEFTyLuN6CnSmucLl26sGfPHnbs2MGnn37KgAEDmD59Oh06dGDfvn3hDi9sSkpKwh3CJaeJWwSw1Yomqn1DRARffiHH31mNN+/oKduK007Mb68k5duhpKy4n9h72lH09iYO9nyVPWnPc2jgIjz/l6ujUZWqGhc8uXhVBeDdeQhEE7eaxuVyUbduXTIyMmjbti0PPPAAX3/9NXl5eYwaVXFw87Rp02jZsiVRUVE0a9aMZ555Bq838EX+iSeeoEWLFpXW/8ADD9C5c+fTbv/jjz+mW7duJCYmkpCQQNeuXfnmm28qtCkoKGDEiBFkZmbidrvJysri2WefDdbv37+fe+65h7S0NKKiomjRogWzZ88G4LPPPkNE2LVrV4V1OhwO/v73vwOwfft2RIR58+Zx2223ERsby5gxYzDGcN9999GkSROio6Np3Lgxo0ePpri4uMK6lixZQpcuXYiJiQm+hy1btvDZZ59ht9vZuXNnhfZz584lISEh7Ec1dQx5hDFFpRhPCYXvrsHZoi7udg2xxbgqtRMRXB0zcHXMIGHSrRR9vJWiRbl43t6I5x9rwS44O2Xgvrkx7psa4boqA1utqDC8I6UilzHGKyJlk4vbgdllk4sDK4wxi8+8hiogOgFvVTn+r7WVypxZybhaZWC8Pgo//q5yfdM0XM3S8BeV4vk0t1K9q0U6zsYp+AuK8Xy5qVJ9bE521QQP1KtXjwEDBjB37lxefvllbDYbY8eO5ZVXXmHKlCm0a9eO3Nxchg0bRlFREU899RSDBg3i2WefZdmyZVxzTWB0cnFxMa+//jrjx48/7bYKCgp48MEHufLKK/F6vUyePJmePXvyww8/kJSUhDGGXr168eOPPzJt2jSys7PZtWsXmzYF9oHH46Fr165ER0czb948GjduzObNmzl06Pzn13/ssceYMGECL774IhA4K5Wamsr8+fNJS0tj7dq1DB06FKfTybhx44BA0nbrrbfyyCOPMH36dNxuN1999RWlpaV069aNZs2aMXv2bP7nf/4nuJ2ZM2fym9/8htjY2POOsSpp4hZhHGkJxP2iI8Xf7qBk4x5Kt+zH3a4h7rb1T7uMRDmJ7t2C6N4tqF3qo+SrHyn+9zaK/72VgglLKXj2SxBwtEjGeXU9XFdl4MxOw9EqBXtKeD+gSlV3Fzq5eFVx1KuN4xRzP6qaqU2bNhw9epQDBw4QFxfHxIkTWbRoET179gSgUaNGPP300wwfPpynnnqK5s2bc8011zB37txg4vbOO+/g8Xj49a9/fdrt/OIXv6jwesaMGbz55pt88MEHDBgwgE8++YTPP/+c5cuX06lT4MqAxo0b87Of/QyA+fPns23bNjZv3kz9+vWD9Rdi6NChDBgwoELZM888E3yelZXFli1b+Mtf/hJM3MaNG0dOTg5TpkwJtmvZsmXw+f3338/UqVMZM2YMNpuNjRs3snTpUl544YULirEqaeIWgcTlIOraJjhbpVO8cjvGEzinb4zBeEpPeQQuuKzTjrtbo8BdF566Cf/RIkr+305Klv9E6fLdFH+0Bc/cNcH2tuQYHK2ScbRKwdGkDvaGtXE0TMDesDa2tFjEpmfblQqnmB5XhDuEy8aZjn6Jw37GeluU88z1ce4qPbp2OmXXQIsI3333HR6Ph1/+8pdIudPpPp+PoqIi8vLySElJYdCgQYwZM4YpU6bgdDqZO3cuffr0oXbt2qfdzrZt23jyySf5+uuv2b9/P36/n8LCQnbsCIxwXrlyJXXq1AkmbSdbuXIlrVu3DiZtF+Pqq6+uVDZz5kxmzZrF9u3bOX78OF6vF3+52RZWrlx5xiOKgwYN4oknnuDDDz8kJyeHWbNm0bFjR9q3b3/R8V4sTdwimD0hhpibWmP8gf+ovj1HKPxoPY76iTibpeHITDxrYmWrFUVUz2ZE9WwGBP7T+3cfpfS7PLy5eZRuyMObe4CiNzfgP3jSCFW3HXtmAvb0OGypsdhTY7GlWc/TYrGlxmKrHYUkRGFLcCNxLk30lFLqEvruu+9ISEggKSmJrVu3ArBw4UKaN29eqW1iYiIA/fr1Y8SIEbz33nvccMMNfPDBB7z11ltn3E6vXr1ITk7mxRdfJDMzE5fLRefOnatscIDN+ltRfjCez+erkHyVOfnU5cKFC3nooYcYP348Xbt2pVatWixcuJAnnnjinLeflJTEr371K2bOnMnNN9/M3Llzefrppy/w3VQtTdwuA2WTb9pqReO6oj6lm/fj3XkIcTtwZCYRdXUjxO08t3WJYK+fgL1+AtzatEKd/2gRvh35eHccwbcjH9+OI/h+zMe37zjeDXmUfLa9cnJXYeUgtdwVk7l4NxLjDDyiHUiME1uME4l2VionyoE47YjLDi474rQFfrrsiNMefI7TFihz2cFpR+yaLKrLkynxUvDWt7jbNcDVvG64w1Fhtnv3bubNm0ffvn2x2Wy0adOGqKgotm7dym233Xba5erUqUPv3r35xz/+wY8//khiYiK33nrradsfPHiQDRs28P777wfb7dq1i/379wfbdOzYkcOHD7NixYpTHnXr2LEjs2fPZteuXac86paamgrATz/9RGZmYOD26tWrTzmrwsm++OIL2rdvzx//+Mdg2fbt2ytt/6OPPmL48OGnXc/QoUO58cYbeemll/B4PPTv3/+s2w4FTdwuI7a4KKI6NcLdIQvv7kN4t+bh238UnIFfc/GG3VDsxV43AXtKPOKwn9/6a0VhaxuFs23aaduYUh/+A4X49hXg338ck1+M/0gR/vyiwPP8Ivz5xRirzL+3AOMpxXi8mMLS4AP/2f9znnvgAg4bYpfADbmtn2KTE8/L1Un5Nicvc7p2Nglsx2ZdKG4TKJvRvnxZ+fJTlZ1jW2xnWMc5tsUmgcGIJ5VVKBep8H4Q64tCWVuhwnM51TJl6yu/zKnWd9IyCDga1cGWoINmTsf4/ZjjxeDTCbdrmpKSEvbu3Yvf7+fgwYMsXbqU5557jtTUVJ577jkA4uLiGD16NKNHj0ZE6N69O16vl3Xr1rFq1SomTJgQXN/AgQO58847yc3NZcCAAdjtp//7UKdOHVJSUpg5cyZNmjTh4MGDjBw5kujo6GCbm266iS5dunDXXXcxadIksrOz+emnn8jNzWXIkCH079+fiRMn0qdPHyZOnEiTJk3YunUrBw4c4K677qJp06Y0bNiQsWPHMnnyZA4cOBB8H2fTokULXn75Zd5++22uuOIK3n33XRYtWlShzZgxY8jJyWHEiBHce++9uN1uvv76a6677rrgKNvOnTvTokUL/vSnPzFw4EDi4+PP63d0qWjidhkSm+DMTMKZGRjZU/ZB9+07ine7NVGngC0hBkeDJKI6ZgHg95QgUc5z+o9x2m077djT47GnX/gH3BgDpb5AElc+oSv2Ykp8UOq3fvowJb6zl5X4Avd89fnBZzDWT3x+jPXzXMuDr0v9mCLviXZ+A4bAaWu/AWOCZfhNoLys7HTlZ2mLMeXWz4n6y1jiW/2Ivr3l2RvWVD7r96/zuNU4X375Jenp6djtdhISEmjVqhUPP/wwDz30UIVTh2PGjCE9PZ3p06fz6KOPEh0dTfPmzRk8eHCF9eXk5JCQkEBubi4LFiw447ZtNhsLFy5k+PDhZGdn07BhQ5599lkee+yxYBsR4b333mP06NEMGzaMgwcPUq9ePYYOHQpATEwMn3/+OSNHjqRfv34UFBSQlZUVnMrE4XDw+uuv8+CDD9K+fXuaN2/O9OnTufHGG8+6b4YOHcq6deu455578Hq99OrVi7Fjx/LII48E29xyyy28//77jB07lpdeegmXy0WHDh2CgyfK3HfffYwYMYL777//rNsNFTmXw46RplOnTmbFiiqbJumyYopL8e47ii/vGP5DBdjio4m6tgkAxxb8B1PqxRYXha1WNBLrxpFRG6c1Ys13qACJciEuB+LQ04/VhTEVkz9jJZLGbwKzRJjAkVC8fjDlklIkcNTVGPyekkByaziRiIogLgcY8B8vssoJrMNvELsdcTkDyxd4Atu14jAGxG7H5nKCAd+x4yfis+IVux2b04Xx+/EfLwy8l7KkFIPYHER1bnTOXwJEZKUx5tRXQkeYc+3D/Mc8FLyxgqjOzXE1O/2RcFVRbm4urVq1CncYKgKMHDmSjz/+mFWrVl30us70uTuf/kuPuNUw4nbibJCEs0FShXJjDO4ODfEfLcJ/zBP4ue8oYrfhbJiM8fo4/na5D64t8EfddUV93G3rY0q8eL783jolabNOK9pwNEjEUbc2pthL6ZZ91imxE6fY7Gm1sCfEBBLK3UesKaFPzEtlT47HFuvG7ynBtzffCjb4D/a0hEB9QTHevUfK1QeeOOolYotx4csvxLf7CIZyR6swOJumYYt24cs7RumuQyeOlhnAGFzZmdiinIHbCv14sFJ91LVNEJeD0q15lG7PC5aX/Yy+uTVit1Gy4adAPQQTG4C43u0AKPp2e+BoqLWcMSAOG3G/6AiA56sf8P54MFiHMdhiXMT17YSIUPjvDXh3HSr33sGWEE1c30A/cPz9Nfj2VZy02ZYcR1zvwAgpz9vf4j9UcVJJe3oCsT0Do+COvbEcc9L9ch0Nkoi5uXWgfsF/MEWlFeqdTVJxdw5cJ3l07leVTuc5W9Ql6vqmGL/h2JylnMx1Rf2LOnJbExjriJvY9YibUlUpPz+f77//nhkzZlSLKUDK08RNAdaEvS3SK5UHj8iKEH1jK0xRSeBUZLEXU+LFVitwTYPx+vEf9VinGf3BnxLnxlG3Nn5PMUXLtlZaf9T1TbEnxOA/WoTn842V6qO7tsDWOBX/4UI8n52ivnsbbLFufAcLKPry+0r1MTltscW48B8ooGjZlkr1jvQ6EO3Cd7CAktU/nrhmy0ouXS3TIcqJP98TSKxOqjc+PwKYohL8+Z5K9ZQ/ol1WZrdZ9xk8UWWLcWGrExtcXiRwXV4Ze0p88HSYlF0b5jrx39fRMLnS8hJ1ot7VMh1/g6QT15UhgQEfFne7BoFbolmxCyDRJ6aVib6uaeCInJVYi02QqBPLR9/c2nqv1vpP2n5s73ZWMn6iHpd1DY1A3F1XB+so2zc6qOSsbDGuwOjxqHMbfKSUOje33347y5Yto1+/ftx9993hDqcCPVWqQsL4TSAxKHc0CmMQtwNxOjBeH/6CoopHrAjMfSRuZ2DQQ4F1xEeC/2CLdSNOO6bUF5zPrny9RLsQhw3j9WO8Xuv6PTlxZM8RSKLKXwuoIlNNPFUK4C8sCYzC1s/vOdNTpSoc9FSpiignH6GpVO+wY699+rs0iNOOvc6Z68UZffp6hw1xnGFiYv2jpyLUmSbcVkpdfvRchFJKqRrncjzbpKqvqvy8aeKmlFKqRnE6nXg8Z5gsXKkq5vF4cDqr5lpUTdyUUkrVKKmpqezevZvCwkI98qYuKWMMhYWF7N69O3g3iIsVsmvcRKQnMBWwA7OMMae8u6uI/BJ4A7jKGLPCKnsc+B3gA4YbYz4MTdRKKaUuN7Vq1QICt1MqLS09S2ulLo7T6SQtLS34ubtYIUncRMQOvAj0AHYBy0VksTFmw0nt4oHfA8vKlbUG+gFtgAxgiYg0N8b4QhG7Ukqpy0+tWrWq7A+pUqEUqlOlVwObjTFbjTElwGvA7ado9xQwASg/0+ftwGvGmGJjzDZgs7U+pZRSSqkaJVSJWz1gZ7nXu6yyIBHpAGQaY94732Wt5e8XkRUisiIvL69qolZKKaWUqkaqxeAEEbEBk4BHL3QdxpgZxphOxphOKSkpVRecUkoppVQ1EarBCbuBzHKv61tlZeKBK4DPrIlQ6wKLRaTPOSyrlFJKKVUjhOqI23KgmYg0EhEXgcEGi8sqjTH5xphkY0yWMSYL+A/QxxpVuhjoJyJuEWkENAO+CVHcSimllFLVRkiOuBljvCLyMPAhgelAZhtjvhOR/wVWGGMWn2HZ70Tkn8AGwAs8dLYRpStXrjwgIjvOI8Rk4MB5tK8uNO7QitS4IXJjP5+4G17KQELpPPuwSP3dQuTGrnGHVk2I+5z7r8vyJvPnS0RWROLNqTXu0IrUuCFyY4/UuEMpkvdRpMaucYeWxl1RtRicoJRSSimlzk4TN6WUUkqpCKGJW8CMcAdwgTTu0IrUuCFyY4/UuEMpkvdRpMaucYeWxl2OXuOmlFJKKRUh9IibUkoppVSEqNGJm4j0FJFNIrJZREaFO57yRCRTRD4VkQ0i8p2I/N4qTxSRj0XkB+tnHatcROQF672stW4hFs747SKySkTetV43EpFlVnyvW/P5Yc3P97pVvkxEssIcd20ReUNENopIrohcFwn7XET+YH1O1ovIAhGJqo77XERmi8h+EVlfruy896+IDLLa/yAig0IVSr3o/AAABmBJREFUf3WjfdgljT/i+jDtv0ISa/j7MGNMjXwQmE9uC9AYcAFrgNbhjqtcfOlAB+t5PPA90BqYCIyyykcBE6zntwH/AgS4FlgW5vj/CMwH3rVe/xPoZz3/G/CA9fxB4G/W837A62GOew4wxHruAmpX931O4N6924Docvt6cHXc58DPgA7A+nJl57V/gURgq/WzjvW8Tjg/N2H6vWsfdmnjj7g+TPuvkMQb9j4sbP8pwv0ArgM+LPf6ceDxcMd1hnjfBnoAm4B0qywd2GQ9fwnoX659sF0YYq0P/Bu4CXjX+tAeABwn73sCkzJfZz13WO0kTHEnWB2InFRerfe51fHttDoBh7XPb62u+xzIOqnTO6/9C/QHXipXXqFdTXloH3ZJY424Pkz7r5DGHNY+rCafKi37sJTZZZVVO9ah4PbAMiDNGLPHqtoLpFnPq9P7mQKMBPzW6yTgiDHGa70uH1swbqs+32ofDo2APOAV6xTJLBGJpZrvc2PMbuB54EdgD4F9uJLI2Odw/vu3Wuz3aiBi9oP2YSGh/Vf4hLQPq8mJW0QQkTjgTWCEMeZo+ToTSNWr1bBgEekF7DfGrAx3LBfAQeAQ+F+NMe2B4wQOewdV031eB7idQMedAcQCPcMa1AWqjvtXXRztw0JG+69qIBT7uCYnbruBzHKv61tl1YaIOAl0ePOMMYus4n0ikm7VpwP7rfLq8n5uAPqIyHbgNQKnGqYCtUWk7N645WMLxm3VJwAHQxlwObuAXcaYZdbrNwh0hNV9n3cHthlj8owxpcAiAr+HSNjncP77t7rs93Cr9vtB+7CQ0v4rfELah9XkxG050MwaueIicJHjaW92H2oiIsDLQK4xZlK5qsVA2QiUQQSuGykrH2iNYrkWyC936DZkjDGPG2PqG2OyCOzTT4wxA4BPgV+dJu6y9/Mrq31YvhEaY/YCO0WkhVV0M7CBar7PCZxiuFZEYqzPTVnc1X6fnyKec9m/HwK3iEgd69v6LVZZTaN92CUQqX2Y9l9hFdo+LJQX9FW3B4ERH98TGJn1RLjjOSm2zgQOt64FVluP2wicy/838AOwBEi02gvwovVe1gGdqsF76MaJEVmNgW+AzcBCwG2VR1mvN1v1jcMccztghbXf3yIw4qfa73NgHLARWA/8A3BXx30OLCBwHUspgSMEv7uQ/Qvca8W/Gbgn3J/1MP7etQ+7tO8hovow7b9CEmvY+zC9c4JSSimlVISoyadKlVJKKaUiiiZuSimllFIRQhM3pZRSSqkIoYmbUkoppVSE0MRNKaWUUipCaOKmaiwRyRIRU26SR6WUigjaf9VcmrgppZRSSkUITdyUUkoppSKEJm6qWhGRDBF5U0TyRGSbiAy3yseKyBsi8rqIHBORb0XkynLLtRKRz0TkiIh8JyJ9ytVFi8ifRWSHiOSLyFIRiS632QEi8qOIHBCRJ0L4dpVSlxHtv1QoaOKmqg0RsQHvAGuAegTuWTdCRG61mtxO4FYnicB84C0RcVo3sn4H+AhIBR4B5pW7Z9/zQEfgemvZkYC/3KY7Ay2s7T0pIq0u2ZtUSl2WtP9SoaK3vFLVhohcAyw0xjQoV/Y40BzYAfQ0xlxrlduA3cCvraYLgQxjjN+qXwBsAv4XOA5ca4xZc9L2soBtQKYxZpdV9g0wyRjz2iV6m0qpy5D2XypUdDSKqk4aAhkicqRcmR34kkDHt7Os0BjjF5FdQIZVtLOs07PsIPCtN5nATYm3nGG7e8s9LwTiLvgdKKVqKu2/VEjoqVJVnewEthljapd7xBtjbrPqM8saWt9Y6wM/WY9Mq6xMAwLfaA8ARUCTkLwDpVRNpf2XCglN3FR18g1wTEQesy7ItYvIFSJylVXfUUT6WvMWjQCKgf8Aywh80xxpXTPSDegNvGZ9i50NTLIuHLaLyHUi4g75u1NKXc60/1IhoYmbqjaMMT6gF9COwLUbB4BZQILV5G3gLuAw8FugrzGm1BhTQqCjy7GW+Qsw0Biz0VruT8A6YDlwCJiAfvaVUlVI+y8VKjo4QUUEERkLNDXG3B3uWJRS6nxo/6WqkmbtSimllFIRQhM3pZRSSqkIoadKlVJKKaUihB5xU0oppZSKEJq4KaWUUkpFCE3clFJKKaUihCZuSimllFIRQhM3pZRSSqkIoYmbUkoppVSE+P98LuqT9F3faQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 可视化观察训练集与验证集的指标变化情况\r\n",
    "def plot(runner, fig_name):\r\n",
    "    plt.figure(figsize=(10,5))\r\n",
    "    epochs = [i for i in range(len(runner.train_scores))]\r\n",
    "\r\n",
    "    plt.subplot(1,2,1)\r\n",
    "    plt.plot(epochs, runner.train_loss, color='#e4007f', label=\"Train loss\")\r\n",
    "    plt.plot(epochs, runner.dev_loss, color='#f19ec2', linestyle='--', label=\"Dev loss\")\r\n",
    "    # 绘制坐标轴和图例\r\n",
    "    plt.ylabel(\"loss\", fontsize='large')\r\n",
    "    plt.xlabel(\"epoch\", fontsize='large')\r\n",
    "    plt.legend(loc='upper right', fontsize='x-large')\r\n",
    "\r\n",
    "    plt.subplot(1,2,2)\r\n",
    "    plt.plot(epochs, runner.train_scores, color='#e4007f', label=\"Train accuracy\")\r\n",
    "    plt.plot(epochs, runner.dev_scores, color='#f19ec2', linestyle='--', label=\"Dev accuracy\")\r\n",
    "    # 绘制坐标轴和图例\r\n",
    "    plt.ylabel(\"score\", fontsize='large')\r\n",
    "    plt.xlabel(\"epoch\", fontsize='large')\r\n",
    "    plt.legend(loc='lower right', fontsize='x-large')\r\n",
    "    \r\n",
    "    plt.savefig(fig_name)\r\n",
    "    plt.show()\r\n",
    "\r\n",
    "plot(runner, 'fw-acc.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.3.4 性能评价\n",
    "\n",
    "使用测试数据对训练完成后的最优模型进行评价，观察模型在测试集上的准确率以及loss情况。代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Test] score/loss: 0.7600/0.4883\n"
     ]
    }
   ],
   "source": [
    "# 模型评价\r\n",
    "runner.load_model(\"best_model.pdparams\")\r\n",
    "score, loss = runner.evaluate([X_test, y_test])\r\n",
    "print(\"[Test] score/loss: {:.4f}/{:.4f}\".format(score, loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从结果来看，模型在测试集上取得了较高的准确率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.4 优化问题\n",
    "\n",
    "在本节中，我们通过实践来发现神经网络模型的优化问题，并思考如何改进。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.4.1 参数初始化\n",
    "\n",
    "实现一个神经网络前，需要先初始化模型参数。如果对每一层的权重和偏置都用0初始化，那么通过第一遍前向计算，所有隐藏层神经元的激活值都相同；在反向传播时，所有权重的更新也都相同，这样会导致隐藏层神经元没有差异性，出现**对称权重现象**。\n",
    "\n",
    "接下来，将模型参数全都初始化为0，看实验结果。这里重新定义了一个类`TwoLayerNet_Zeros`，两个线性层的参数全都初始化为0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.nn as nn\n",
    "import paddle.nn.functional as F\n",
    "from paddle.nn.initializer import Constant, Normal, Uniform\n",
    "\n",
    "class Model_MLP_L2_V4(paddle.nn.Layer):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(Model_MLP_L2_V4, self).__init__()\n",
    "        # 使用'paddle.nn.Linear'定义线性层。\n",
    "        # 其中in_features为线性层输入维度；out_features为线性层输出维度\n",
    "        # weight_attr为权重参数属性\n",
    "        # bias_attr为偏置参数属性\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size,\n",
    "                                weight_attr=paddle.ParamAttr(initializer=Constant(value=0.0)),\n",
    "                                bias_attr=paddle.ParamAttr(initializer=Constant(value=0.0)))\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size,\n",
    "                                weight_attr=paddle.ParamAttr(initializer=Constant(value=0.0)),\n",
    "                                bias_attr=paddle.ParamAttr(initializer=Constant(value=0.0)))\n",
    "        # 使用'paddle.nn.functional.sigmoid'定义 Logistic 激活函数\n",
    "        self.act_fn = F.sigmoid\n",
    "        \n",
    "    # 前向计算\n",
    "    def forward(self, inputs):\n",
    "        z1 = self.fc1(inputs)\n",
    "        a1 = self.act_fn(z1)\n",
    "        z2 = self.fc2(a1)\n",
    "        a2 = self.act_fn(z2)\n",
    "        return a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def print_weights(runner):\n",
    "    print('The weights of the Layers：')\n",
    "    \n",
    "    for item in runner.model.sublayers():\n",
    "        print(item.full_name())\n",
    "        for param in item.parameters():\n",
    "            print(param.numpy())\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "利用Runner类训练模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The weights of the Layers：\n",
      "linear_2\n",
      "[[0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]]\n",
      "[0. 0. 0. 0. 0.]\n",
      "linear_3\n",
      "[[0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]]\n",
      "[0.]\n",
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.45000\n",
      "[Train] epoch: 0/5, loss: 0.693146824836731\n",
      "The weights of the Layers：\n",
      "linear_2\n",
      "[[0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]]\n",
      "[0. 0. 0. 0. 0.]\n",
      "linear_3\n",
      "[[-0.00203125]\n",
      " [-0.00203125]\n",
      " [-0.00203125]\n",
      " [-0.00203125]\n",
      " [-0.00203125]]\n",
      "[-0.0040625]\n",
      "The weights of the Layers：\n",
      "linear_2\n",
      "[[-2.2261411e-05 -2.2261411e-05 -2.2261411e-05 -2.2261411e-05\n",
      "  -2.2261411e-05]\n",
      " [ 2.1618824e-05  2.1618824e-05  2.1618824e-05  2.1618824e-05\n",
      "   2.1618824e-05]]\n",
      "[1.8309125e-06 1.8309125e-06 1.8309125e-06 1.8309125e-06 1.8309125e-06]\n",
      "linear_3\n",
      "[[-0.00383399]\n",
      " [-0.00383399]\n",
      " [-0.00383399]\n",
      " [-0.00383399]\n",
      " [-0.00383399]]\n",
      "[-0.00766798]\n",
      "The weights of the Layers：\n",
      "linear_2\n",
      "[[-6.445288e-05 -6.445288e-05 -6.445288e-05 -6.445288e-05 -6.445288e-05]\n",
      " [ 6.231440e-05  6.231440e-05  6.231440e-05  6.231440e-05  6.231440e-05]]\n",
      "[4.897984e-06 4.897984e-06 4.897984e-06 4.897984e-06 4.897984e-06]\n",
      "linear_3\n",
      "[[-0.0054344]\n",
      " [-0.0054344]\n",
      " [-0.0054344]\n",
      " [-0.0054344]\n",
      " [-0.0054344]]\n",
      "[-0.01086785]\n",
      "The weights of the Layers：\n",
      "linear_2\n",
      "[[-0.00012447 -0.00012447 -0.00012447 -0.00012447 -0.00012447]\n",
      " [ 0.00011986  0.00011986  0.00011986  0.00011986  0.00011986]]\n",
      "[8.756236e-06 8.756236e-06 8.756236e-06 8.756236e-06 8.756236e-06]\n",
      "linear_3\n",
      "[[-0.00685571]\n",
      " [-0.00685571]\n",
      " [-0.00685571]\n",
      " [-0.00685571]\n",
      " [-0.00685571]]\n",
      "[-0.01370772]\n"
     ]
    }
   ],
   "source": [
    "# 设置模型\n",
    "input_size = 2\n",
    "hidden_size = 5\n",
    "output_size = 1\n",
    "model = Model_MLP_L2_V4(input_size=input_size, hidden_size=hidden_size, output_size=output_size)\n",
    "\n",
    "# 设置损失函数\n",
    "loss_fn = F.binary_cross_entropy\n",
    "\n",
    "# 设置优化器\n",
    "learning_rate = 0.2 #5e-2\n",
    "optimizer = paddle.optimizer.SGD(learning_rate=learning_rate, parameters=model.parameters())\n",
    "\n",
    "# 设置评价指标\n",
    "metric = accuracy\n",
    "\n",
    "# 其他参数\n",
    "epoch = 2000\n",
    "saved_path = 'best_model.pdparams'\n",
    "\n",
    "# 实例化RunnerV2类，并传入训练配置\n",
    "runner = RunnerV2_2(model, optimizer, metric, loss_fn)\n",
    "\n",
    "runner.train([X_train, y_train], [X_dev, y_dev], num_epochs=5, log_epochs=50, save_path=\"best_model.pdparams\",custom_print_log=print_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "可视化训练和验证集上的主准确率和loss变化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnoAAAFDCAYAAACtN6P0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xl8XHW9//HXZyYzyUy6723a0i0pLVBLCRXZWtRCuUJRuCCIYrlAQcUFuSro1Z9XREERWUQpsrizCF4tOyiyyNqU0tKN0n1vQykNnUySWT6/P2ZS09AlbZOZZPJ+Ph7z6JzvOWfO5yTNzGe+q7k7IiIiIlJ4AvkOQERERETahhI9ERERkQKlRE9ERESkQCnRExERESlQSvRERERECpQSPREREZECpURPREREpEAp0RMREREpUEr0RERERApUUb4DaC/69Onjw4YNy3cYIpJDc+bMecfd++Y7joOl9y+Rzqel719K9LKGDRtGVVVVvsMQkRwys9X5jqE16P1LpPNp6fuXmm5FRERECpQSPREREZECpURPREREpEAp0RMREREpUEr0RERERAqURt1Kp1ZTU8OWLVtIJBL5DkVaWSgUol+/fnTr1i3foYiI5I0SPem0ampq2Lx5M2VlZUQiEcws3yFJK3F34vE469evB1CyJyKdlppupdPasmULZWVlRKNRJXkFxsyIRqOUlZWxZcuWfIcjIpI3SvSk00okEkQikXyHIW0oEonkrVnezKaa2VtmtszMrtrN/ulmVm1mb2QfF2fLx5vZy2a20Mzmm9mncx+9iBQKNd1Kp6aavMLhqTQW3PW7a75+v2YWBG4DpgDrgNlmNsvdFzU79H53v7xZWS1wgbu/bWaDgDlm9qS7v9f2kRcGd6fu/xaT3hrPdygi+6X4lJEUDe3Rqq+pRE9EOjRPp/F4Ao83EOgWwcLt4m1tIrDM3VcAmNl9wBlA80TvA9x9aZPnG8xsC9AXUKLXQg3PruLdsx7Idxgi+63Xw+cp0RMRgUytjdcl8NoGcMdKQhBsN71RyoC1TbbXAR/ezXFnmdmJwFLgCndveg5mNhEIA8ubn2hmM4AZAEOHDm2lsAtD7PYqrFeEfnNmYKFgvsMRabFAr9bvTqRET0R2sWTJEsaMGcPs2bOprKw84Nd54oknOPXUU6murqZPnz6tGGFG+v06aEhCKEigtBgr6nAf6A8D97p7vZldCvwW+GjjTjMbCPwe+Ly7p5uf7O53AHcAVFZWem5Cbv9Sm3cQ/8tiSr88kaJhPfMdjkjetZuvvyLSMma218ewYcMO6vXLy8vZuHEj48ePb52AW5EnU3g6k9MEIiGsayTTXNv+krz1wJAm24OzZTu5+1Z3r89u3gkc1bjPzLoBjwLfcfdX2jjWglJ7z1xIpim99MC/pIgUEtXoiXQwGzdu3Pn8pZde4qyzzuL1119n4MCBAASDu096GhoaCIfD+3z9YDDIgAEDWifYVuLpNF7bgNclsEgYKy3GQkW046E0s4FyMxtOJsE7F/hM0wPMbKC7N/4ypwGLs+Vh4P+A37n7g7kLuePzdJrYHXMITx5GaHTr1yKLdESq0RPpYAYMGLDz0atXLwD69u27s6xv3747j/vf//1fZsyYQa9evZgyZQoAN9xwA+PGjaO0tJRBgwbx2c9+dpe55pYsWYKZUVVVtcv2X/7yF0499VSi0SijRo3iT3/6037H/q9//Yvjjz+ekpISevXqxQUXXMDWrVt37l+9ejWf/OQn6d27N5FIhFGjRnHTDT8jva0Wr0vw0BMPc+SxE4lGo/Ts2ZOPfOQjLFiw4IB/lm3F3ZPA5cCTZBK4B9x9oZn9wMymZQ/7SnYKlXnAV4Dp2fJzgBOB6U2mXml/1avtUP3TK0itfI/SS4/a98EinYRq9ESaeO9rj5N4Y1POrxsaP4AeN53a6q/7s5/9jKuuuopXX32VVCoFZJp+b7rpJoYPH86GDRu44oor+NznPseTTz6519f61re+xfXXX8+tt97K7bffzvTp0zn22GNb3FS8du1aTjnlFM4++2xmzpzJO++8w2WXXca5557L008/DcAll1xCMBjkmWeeoXv37ry9YDFbN22BoiDrtm7iM9Mv4Gc/+xnTpk0jHo8zZ86cPdZg5pu7PwY81qzse02eXw1cvZvz/gD8oc0DLECxmVUE+kaJfGpMvkMRaTeU6IkUsBNOOIFvf/vbu5RdeeWVO58PHz6cm2++mWOPPZatW7fSu3fvPb7WFVdcwZlnngnAj370I2699Vaee+65Fid6t9xyC/379+fOO++kqCjz1vOb3/yGY445htdee42JEyeyevVqpn/+84w7/AgsGOCQwUPwtBMIF7Fh0SbS6TTnnHPOzmbqsWPH7s+PQwpYakMNdbPeosuVx2LF+mgTaaS/BpEm2qJWLZ8mTpz4gbK///3vXH/99SxZsoT33nuPdDozoHP16tV7TfSaDs4Ih8P06dOHzZs3tziWhQsXcuyxx+5M8hrjKykpYeHChRxdWcnXvvRlvnLlFcz669+Y/LGPctppp3HccccBcPTRRzNp0iRGjx7NlClTmDx5MmeeeSZlZWUtjkEKV+yuuZBySmeo2VakKfXREylgpaWlu2wvW7aM0047jdGjR3P//fdTVVXFn//8ZyAzWGNvmg/kMLOdSeLBStcnSW+rZcZnPs/yhUv4r4svYs2aNUyZMoWLL74YgKKiIp555hmeeuopjjzySO677z7Ky8t3NvtK5+WpNLW/nkPxlBEUjeyV73BE2hUleiKdyKuvvkoikeCmm27i2GOPZfTo0WzalJs+iYcddhgvvfQSyWRyZ9lrr71GXV0dh40sz8yH1yPK0IqRXDJjBn/84x/55S9/yd133019fWYWEjPjmGOO4X/+53948cUXmThxIr/5zW9yEr+0X3WPv01qbY2mVBHZjZwlevta4Dt7zDlmtig7Eu1PTcqvN7MF2cenm5T/xsxWNh+ZZhm3ZK8138wmtP0dirR/FRUVpNNpfv7zn7Ny5UoeeughfvzjH+fk2l/96lfZvHkzF198MQvemM+z/3iGCy+8kI9/7GN8eNLxBLtF+MLlX+KJJ55g+fLlLFiwgL/+9a+MHDmS4uJinn32WX70ox/x2muvsWbNGp566ikWLVqkfnpC7cw5BAZ0oWTa6HyHItLu5CTRa7LA96nAWOA8Mxvb7JhyMiPQjnP3w4CvZcs/AUwAxpNZQui/s5OJNvqGu4/PPt7Ilp0KlGcfM4BftdnNiXQgRx99NDfeeCM333wzY8eO5dZbb+XnP/95Tq49ePBgnnjscd5e8haVx0zkzP/8TyorK7nv/vt3rk+bSqX48pe/zOGHH86kSZNIpVI8/PDDAPTs2ZPnn3+e008/nfLycmbMmMFFF13Et771rZzEL+1Tcs171D32NqUXHanlzkR2w9zbfuUcM/sI8H13PyW7fTWAu/+4yTE/AZa6+53Nzv0GUOLu12S37wKedPcHzOw3wCPNJxU1s5nAs+5+b3b7LWByk8lJP6CystIb5w2TzmHx4sWMGaNpGHLB3fF4Ax5PZNaljYQzj0DbT3m8t9+zmc1x9w7f3teZ379qvvcM7//wefqv/BpFh7TuYvAi7VlL379y1XS7uwW+mw+VqwAqzOxFM3vFzKZmy+cBU80samZ9gJPYdWmha7PNsz83s+L9uB5mNsPMqsysqrq6+sDvTkT2yusSeG1Dph9ez9LM2rQ5SPKksHkyReyuuRSfWq4kT2QP2tNgjCIyTa2TgfOAX5tZD3d/isykoy8B9wIvA6nsOVcDhwJHA72A/WrDcfc73L3S3SsbVxMQkdbhiRTphszACysJEegeIdgtggXb09uOdGR1jywlveF9rYQhshe5esfd5wLfZGrdZrl7wt1XAkvJJH64+7XZPnhTAMvuw903ekY9cA/QOGlYS64nIm3AU2lS78dJb6/NrE/rjplhIU3bKa0rdnsVgbKulPxHeb5DEWm3cpXo7VzgO7tg97nArGbH/JVMbR7ZJtoKYIWZBc2sd7Z8HDAOeCq7PTD7rwGfBBoXvZwFXJAdfXsMsH1v/fNE5OB52knH6klvq4X6FBYJE+gWIfPnKdK6kiu3Uf/UckovOQor0iAMkT3JyVdsd0+aWeMC30Hg7sYFvoEqd5+V3XeymS0i0zT7DXffamYlwAvZD4sa4LPZBcMB/mhmfcnU8r0BXJYtfwz4D2AZUAtcmIv7FOnMPJHC4w1QXEQgWqwmWmlTsV/PATNKLzoy36GItGs5a0tpwQLfDnw9+2h6TB2ZKVl295of3UO5A186yJBFZB88kcRTTqAkhIWDWI+oalekzXlDktq75lJyegXBwd3zHY5Iu6ZOMyKy3zyVJh2rh4YkBAN4cVGmiVZJnuRA3d/eIr0lppUwRFpAiZ6ItJinG+fDawAzLJqdD0/98CSHYrdXETykO8Unj8x3KCLtnjrRiEjLpdJ4vAErDhHoEc30xVOSJzmUfHsr9c+spHTGUeoHKtIC+isRkb3yhiTp2noArHHC464lXHjRf/Hxj388z9FJZxO7Yw4UBYj+lwZhiLSEEj2RDmj69OmZuenMCIVC9OnTh+OPP56f/OQnxGKxVrmGJ1OkauKka+J4XRJPZ5ZLVC2K5IvXJai9Zy4lZ4wmOKBrvsMR6RD0ji3SQZ1wwgls3LiR1atX889//pPzzz+fX/ziF0yYMIHNmzcf8Ot62knvqCP9Xi0kUli0mEDPqJYsk7yL/2Ux6a1xSi/TIAyRllKiJ9JBhcNhBgwYwKBBgzjiiCP4whe+wMsvv0x1dTVXXXXVLsfeeuutHHrooZSUlFBeXs61115LMpmZjvI73/kOo0eP/vfB7nh9ki9ddSWTzphKINqywRbuzg033MCIESMIh8OMHDmSm266aZdj/va3v3HkkUcSjUbp0aMHEydOZO7cuQAkEgm+/vWvM3jwYIqLixk4cCDnnnvuQf6UpJDEZs4hOLInxR8dnu9QRDoMjboVaSb2+PwPlIWG9SE8ZhCeTFH79MIP7h/Vn3B5f9J1CeL/XPyB/eHRAwmN6Et6Rz3xF976wP7SU8e1SuxlZWWcf/75/O53v+Ouu+4iEAjw/e9/n3vuuYebbrqJ8ePHs3jxYi677DLq6uq45ppruOCCC/jRj37Ey/98nmMmn4AFAySiQR74y4Ncd911Lb72L3/5S7773e9y8803c9JJJ/GPf/yDr33ta3Tt2pWLLrqITZs2cfbZZ/PDH/6Qs88+m7q6OubOnUtRUeZt6NZbb+WBBx7gD3/4AyNGjGDz5s28+OKLrfJzkY4vsWgLDc+vptv1H8cCqqMQaSkleiIF5rDDDqOmpoZ33nmHLl268JOf/IS//OUvTJ06FYDhw4fzwx/+kK985Sv84P99n1EDhjDxqEp+/8c/cMyk48GMRx59lHg8zjnnnNPi61533XV8+ctfZsaMGQCUl5fz1ltvce2113LRRRexceNGEokE55xzDsOGDQNgzJgxO89fvXo1FRUVTJo0CTNj6NChHH300a33g5EOLXbHHAgFiF6oQRgi+0OJnkgze6tds6LgXvcHSkJ739+luNVq7/YkszAMmBkLFy4kHo9z1lln7dL8mkqlqKurY/OyNfTt25cLPncB/++a/+XmX91GKBDgd7/7HdOmTaNHjx4tumZNTQ3r1q3jxBNP3KV80qRJ3HzzzdTW1jJu3DhOOeUUDj/8cKZMmcLkyZM588wzGTJkCAAXXnghU6ZMYdSoUUyZMoUpU6Zw+umnEw6HW+knIx2VxxPU/nYekbPGEuxbmu9wRDoU1X+LFJiFCxfSvXt3evfuTTqdBuDPf/4zb7zxxs7H/HnzWTJ7Hr0H9iPQM8pnLvgs77//Po8++ijV1dU88cQTfP7zn2/VuILBII8//jjPPPMMRx99NA899BAVFRU88sgjAIwfP56VK1dyww03EA6H+epXv8r48eOpqalp1Tik46l9YCH+Xp0GYYgcACV6IgVk/fr1/PGPf+TMM88kEAhw2GGHUVJSwvLlyxkx5BCG9y1j5MiRlFeUU3HUEYS6l2KBAD179uT000/n97//Pffeey+9evXilFNOafF1u3XrxuDBg3n++ed3KX/uuecYPnw40WgUyNQyTpw4kW9/+9s8//zzTJo0iXvuuWfn8V26dOFTn/oUt9xyC1VVVSxevJjnnnuudX440mHVzqyi6NA+hE88JN+hiHQ4aroV6aAaGhrYtGkT6XSarVu38q9//Ysf//jH9OvXjx//+MdAJnG6+qqr+M53vgO1DXzspI+SLiliwaKFzJ07l+uvv37n611wwQWcffbZLF68mPPPP59gcP/Wrb366qu58sorKS8vZ/LkyTzzzDP86le/4rbbbgPgpZde4h//+Acnn3wyAwcO5O2332b+/PlcdNFFAPz0pz9l0KBBjB8/nmg0yr333kswGKSioqKVfmLSESXmb6Lh5XV0v/EUrcIicgCU6Il0UC+88AIDBw4kGAzSvXt3xowZw+WXX86XvvQlSktLcXe8toHvfOnrDOjem1/e/Wu+8f++QyQSoaKigunTp+/yeqeeeirdu3dn8eLF3Hvvvfsdzxe+8AVisRg/+tGP+OIXv8iQIUO47rrrdiZy3bt35+WXX+a2225j27ZtDBgwgPPPP5/vfve7QKZW8MYbb+Ttt98mnU4zZswYHnrooV2nfulAzGwqcDMQBO509+ua7Z8O/BRYny36hbvfmd33BHAM8C93Py1nQbdDsZlzoDhI9PMfyncoIh2SNXbc7uwqKyu9qqoq32FIDi1evHiXUZ+Fxt1Jb49jwQBWGu60U1Ls7fdsZnPcvdU7fplZEFgKTAHWAbOB89x9UZNjpgOV7n75bs7/GBAFLm1Joleo71/pHfVsGvQzSj55KL1+d2a+wxFpV1r6/tU53/lFCpSn0qR31OHpNGZGoHuEQNeSTpvk5dFEYJm7r3D3BuA+4IyWnuzu/wDeb6vgOor4fQvw9xsovVSDMEQOlN79RQqAu5OON5B+rxavS+CJzGhb9WnKmzJgbZPtddmy5s4ys/lm9qCZDclNaB1HbOYcig7vR/hY/WhEDpQSPZEOzpMp0tvjeKweigIEekQJFKv7bQfwMDDM3ccBTwO/3Z+TzWyGmVWZWVV1dXWbBJhPDXM2kKjaQOmlR+kLi8hBUKIn0sGlaxsglca6lBDoFsGK9m+0rLSJ9UDTaqjB/HvQBQDuvtXd67ObdwJH7c8F3P0Od69098q+ffseVLDtUWxmFRYpIvrZtp1gXKTQKdGTTq2jDkZKNyTxZArIrLYR6BklUBJSzUczefz9zgbKzWy4mYWBc4FZTQ8ws4FNNqcBH1wkuZNK19QR/9ObRM47gkCPSL7DEenQ1L4jnVYoFCIej++czLcj8FSadKweGpJQHMK6BjXQYi/i8TihUCjn13X3pJldDjxJZnqVu919oZn9AKhy91nAV8xsGpAE3gWmN55vZi8AhwJdzGwdcJG7P5nr+8iX2j++iccSlF66X5WcIrIbSvSk0+rXrx/r16+nrKyMSCTSrmvD3D0zyKK2AdyxaBiLaA3YPXF34vE469evp3///vmK4THgsWZl32vy/Grg6j2ce0LbRtd+uTu1M6sIHTmA0NG7G78iIvtDiZ50Wt26dQNgw4YNJBKJPEezd55I4vVJCAaw4hAWaL9JaXsRCoXo37//zt+zdAyJ19aTmLeZHref1q6/fIl0FEr0pFPr1q1bu00EPJEiXVtPsHsUT6RIrnuXomF99OEnBS12exXWJUzkM0fkOxSRgqBET6QdSqzZSt0ry7GiAKWfPAoLBQkNL7yRlSJNpd+LE79/AZELPkSga3G+wxEpCEr0RNqRdKyeuleWk1yzlUCPKCXHjlIzrXQatb+fj8eTWglDpBUp0RNpJ1Lv7iD26Hxwp/ioYYQPL9OIWuk03J3Y7VWEjh5E+MiB+z5BRFpEiZ5InnlDEgsXEehRSnj0AMJjBhHoWpLvsERyquHFNSQXVdPjrmn5DkWkoKi6QCRPvCFJ3SvL2fFQFem6BBYwSiaOUJInnVLs9iqsWzGRTx+e71BECopq9ERyzN1Jrt5K3avL8doGQocOVD886dRSW2uJP7iI0osnECjV/JAirUmJnkgOeTJN/NnFJNe+S6BXKZGPjqGob/uc3kUkV2p/+wbUpzQIQ6QNKNETyQF3x8ywogAWLqL46OGEx5apJk86vcxKGHMIHzuE0BH5WcVEpJCpj55IG0tVv0/to/NIba8FIHLiaIoPH6wkTwRoeHYVyaVbKb1MtXkibUE1eiJtxBuS1M1ZRWLJRiwaxuMN0D2a77BE2pXYzCqsZwmR/xyb71BECpISPZE2kFhVTd0ry/G6BOExgyiecAgW1p+bSFOpLTuI/2UxpV+aiEVC+Q5HpCDpk0ekDaS2vI9Fi4l+/DCCfbrmOxyRdqn2njcgkab00qPyHYpIwVKiJ9IKPJ2mYcF6gv26UjSgB8UThlEcMPXDE9kDT6eJ3TGH8KRDCB2qdZxF2ooGY4gcpOTmGmJ/m0v9nFUk17wLkBldqyRPZI/q/76C1IptmlJFpI2pRk/kAHl9grqqVSSWbsJKi4l8bCyhob3zHZZIhxCbOYdAnyiRM8fkOxSRgqZET+QAJVZWk3h7E+HDyig+8hAsFMx3SCIdQmpDDXV/W0KXr38EK9bHkEhbylnTrZlNNbO3zGyZmV21h2POMbNFZrbQzP7UpPx6M1uQfXx6N+fdYmY7mmxPN7NqM3sj+7i4be5KOpt0TZzk+m0AhCoGUnrGBEomjlCSJ7IfYnfPhZRTOkODMETaWk6+SplZELgNmAKsA2ab2Sx3X9TkmHLgauA4d99mZv2y5Z8AJgDjgWLgWTN73N1rsvsrgZ67uez97n55W96XdB6eStOwYB3189Zg0WK6nFmJBYxgz9J8hybSoXgqTe2vX6f44yMoGqWuDiJtLVc1ehOBZe6+wt0bgPuAM5odcwlwm7tvA3D3LdnyscDz7p509xgwH5gKOxPInwLfzME9SCeV3LSd2N9ep/711RQN6U3pf4zTQAuRA1T/xDJSa7ZrShWRHMlVolcGrG2yvS5b1lQFUGFmL5rZK2Y2NVs+D5hqZlEz6wOcBAzJ7rscmOXuG3dzzbPMbL6ZPWhmQ3azX2SfUu/uoPbx+XgqTeTjhxE9aQyBaHG+wxLpsGIzqwgM6ELJGYfmOxSRTqE99YItAsqBycBg4HkzO8LdnzKzo4GXgGrgZSBlZoOAs7PHN/cwcK+715vZpcBvgY82P8jMZgAzAIYOHdrqNyQdk7uT3lZLsFcpwV5dKDmhgtAhfdQPT+QgJddup+7Rt+ly1fH6exLJkVzV6K3n37VwkEnk1jc7Zh2Z2rmEu68ElpJJ/HD3a919vLtPASy770hgFLDMzFYBUTNblj1+q7vXZ1/3TmC3bQTufoe7V7p7Zd++mrBTILW9lton3yT28FzS78cBCI/qrw8lkVZQe+fr4E7pJRPyHYpIp5GrGr3ZQLmZDSeT4J0LfKbZMX8FzgPuyTbRVgArsv3werj7VjMbB4wDnnL3JDCg8WQz2+Huo7LPBzZpzp0GLG7De5MC4Kk0DfPXUj9/LRQFKPnwSKxLSb7DEikYnkwRu/N1iqeOomjY7sbPiUhbyEmi5+5JM7sceBIIAne7+0Iz+wFQ5e6zsvtONrNFQAr4Rja5KwFeMDOAGuCz2SRvb75iZtOAJPAuML1NbkwKgqfSxGbNJf1eLUUj+lJy9AgC0XC+wxIpKHWPLCW94X1Kf/mJfIci0qnkrI+euz8GPNas7HtNnjvw9eyj6TF1ZEbe7uv1uzR5fjWZqVpE9shTaSwYwIIBQqP6E+xVSlGZahqkdWQHlN1M5svtne5+XbP908nMGtDYjeUX7n5ndt/ngf/Jlv/Q3X+bk6DbUGzmHAJlXSn5RHm+QxHpVNrTYAyRnEmu30b8xbeJnFBB0cAeFB8xON8hSQFpydyhWR+Y79PMegH/D6gEHJiTPXdbDkJvE8mV26h/chldvzcJK1J/V5FcytnKGCLtgSdSxF9eRu1TC7CigAZZSFtpydyhe3IK8LS7v5tN7p4mO3doRxX79Rwwo/RiDcIQyTXV6Emnkdy8nfgLS/H36zLr0044RLUL0lZ2N3foh3dz3FlmdiKZmQSucPe1ezi3+byjHYYnUtTePZeS0yoIDu6e73BEOh3V6Emnkd66Axyipx6RWZ9WSZ7k18PAMHcfR6bWbr/64ZnZDDOrMrOq6urqNgmwNdT9bQnpzTGthCGSJ0r0pKCl3nmfxNqtAITGDKLLJydQNKBHnqOSTmCfc4fuZb7Plsw72mHmAY3dXkVwaHeKTxmV71BEOiUlelKQPJ2mfu5qYo/Mo75qFe6OmalPnuTKzrlDzSxMZu7QWU0PMLOBTTabzvfZONVUTzPrCZycLetwksu2Uv+PlZTOOAoL6uNGJB/UR08KTmpbjPgLS0lv3UFoRF9KjhlJdh5GkZxo4dyhu53v093fNbNryCSLAD9w93dzfhOtIHbHHCgKEP2vI/MdikinpURPCkq6Jk7s4blYUZDISWMIDeuT75Ckk2rB3KF7nO/T3e8G7m7TANuY1yepvecNSqaNJjiwa77DEem0lOhJQfBECgsFCXSLUFw5nNDwvgQiWt1CJF/if1lM+p1aSi+rzHcoIp2aOk1Ih+buNCzZwI4/v0ZqWwyA4rFlSvJE8ix2exXBET0p/tjwfIci0qmpRk86rPSOeuIvLiW14T2Cg3pgIf13FmkPEouraXh+Nd2u+zgWUH2CSD7pk1E6pIblW6h7ZRmknZKPjCQ0eqAGXIi0E7E75kAoQPTC8fkORaTTU6InHVL63RjBHqVETqgg0C2S73BEJMvjCWp/8waRM8cQ7Ncl3+GIdHpK9KTDSKx6ByspomhAD4onHAJmWEC1eCLtSfzPC/H36jQIQ6SdUKIn7Z7XJ6h7ZTmJFdUUDe1N0YAemnxVpJ2KzZxD0ejehCcNy3coIoISPWnnkuveJf7i23i8gfD4oRR/aMi+TxKRvEi8uZmGl9bS7Wcnq8+sSDuhRE/areSG96h9eiGBHlEwv1wLAAAgAElEQVSiHxtLsI8mXRVpz2Izq6A4SPTzGoQh0l4o0ZN2x+sTWHGI4MDumRG1owZgRWqqFWnP0rEGan8/n8jZhxHsHc13OCKSpU9PaTc8maLutRXseKiKdKweMyN86CAleSIdQPy+BXhNPaWXHpXvUESkCdXoSbuQqn6f+Atvkd4eJ3ToQCys/5oiHUlsZhVFh/UlfNzQfIciIk3o01Tyyt2pn7uahvlrsUiY6MmHU1TWM99hich+aHh9A4nZG+h+y6kahCHSzijRk7wyM/z9OkIj+lHy4ZFYsf5LinQ0sZlzsEgR0c+Ny3coItKMPlUl5zztNCxcR9HgXgR7llJyQoXWwxTpoNLv1xP/05tEzj2cQA+tUiPS3ijRk5xK18SJv7CU1JYavCFF8KhSJXkiHVj8j/PxHQ2UXqqVMETaIyV6khPuTmLJRuqqVkIgQOTE0RSN6JvvsETkILg7sdurCI0fQGhiWb7DEZHdUKInOZFYuom6V5YTLOtJ5LhyAqXF+Q5JRA5S4rX1JOZtpsevPqFBGCLtlBI9aTPujscTBKJhQqP6Y0VBikb01QeCSIGIzazCSkNEPnNEvkMRkT1Q5yhpE+naBuL/WETs0TfwRBILBgiN7KckT6RApN+LE79vAZHzxxHoVpLvcERkD1SjJ60usaqaupeW4ckUxROGQVEw3yGJSCur/f18PJ7UShgi7ZwSPWk1nkwRf/FtkiuqCfTpQvSE0QR7aM1LkULj7sRmVhGqHER4wqB8hyMie6FET1pPMIDXJyk+8hDC4wZr2hSRAtXw0lqSC6vpcee0fIciIvugRE8OiieS1M9ZTXjcYALRYqJTDlM/PJECF7u9CutWTOTcw/MdiojsgxI9OWDJje8R/9dSfEc9gT5dCI/qryRPpMClttYS//NCSi+aQKA0nO9wRGQflOjJfvNkivo5q2hYtIFA1xIi//Ehivp3y3dYIpID8d/Ng/qUBmGIdBBK9GS/1c9dQ8OiDYTGDKTkqOFYSKNqRTqDxkEY4Y8MJjRuQL7DEZEWUG95aRFPpUnH6gEoHjeY6CmHEzlmlJI8kT0ws6lm9paZLTOzq/Zy3Flm5mZWmd0Om9k9Zvammc0zs8k5C3ofGp5bRfKtrZRepnVtRToK1ejJPqXejRF/4S0wo/S08VhxiKJBPfMdlki7ZWZB4DZgCrAOmG1ms9x9UbPjugJfBV5tUnwJgLsfYWb9gMfN7Gh3T+cm+j2L3V6F9SwhcvZh+Q5FRFpINXqyR5526uevJfbwXDzeQPH4oVhAgy1EWmAisMzdV7h7A3AfcMZujrsGuB6oa1I2FngGwN23AO8Bea9CS23ZQfwvi4le8CEsEsp3OCLSQkr0ZLfSsXpqH5tH/ZxVFA3tTeknjyI0tHe+wxLpKMqAtU2212XLdjKzCcAQd3+02bnzgGlmVmRmw4GjgCHNL2BmM8ysysyqqqurWzf63aj9zRuQSFN6ad5zThHZD2q6ld2y4iIIBohMGk3R8L6aNkWkFZlZALgRmL6b3XcDY4AqYDXwEpBqfpC73wHcAVBZWeltFSuAp9PEZs4hfOIhhMb0bctLiUgrU6InO3kyRf28tRSPG4yFiohOPUIJnsiBWc+utXCDs2WNugKHA89m/8YGALPMbJq7VwFXNB5oZi8BS9s84r2o/8dKUiu20e2ak/IZhogcgJw13bZkBJqZnWNmi8xsoZn9qUn59Wa2IPv49G7Ou8XMdjTZLjaz+7PXetXMhrXFPRWS1LYYsYffoGH+WpLrtgEoyRM5cLOBcjMbbmZh4FxgVuNOd9/u7n3cfZi7DwNeAaa5e5WZRc2sFMDMpgDJ5oM4ci02s4pAnyiRs8bmMwwROQA5qdFryQg0MysHrgaOc/dt2dFmmNkngAnAeKCYzDfgx929Jru/Emg+BPQiYJu7jzKzc8l0dv5AgiiZebESb2+m7pXlWChI9OTDKSrTiFoRy3zTuRg4D+jj7uPM7ERggLs/sLdz3T1pZpcDTwJB4G53X2hmPwCq3H3WXk7vBzxpZmkytYCfa437OVCpje9T99cldLniI5kuHSLSoeTqr3bnCDQAM2scgdb0W+olwG3uvg12jjaDzAi05909CSTNbD4wFXggm0D+FPgM8Kkmr3UG8P3s8weBX5iZuXub9mPpiBreXEf9nFUEB3YncuKhBKJa0kgk6wdkvpzeBNyeLVsH/BzYa6IH4O6PAY81K/veHo6d3OT5KmD0gQTcFmrvngspp3SGVsIQ6Yhy1XS7zxFoQAVQYWYvmtkrZjY1Wz4PmJptzugDnMS/+75cDsxy9417ul42QdwOfGDIaK5HrbUnjTlvaEQ/io8aRvTkI5TkiexqOnCau98HNH5JXAmMyFtEOeapNLE75lD8seEUlWvUvUhH1J7q4YuAcmAymY7Lz5vZEe7+lJkdTWbkWTXwMpAys0HA2dnjD0guR621F+5OYslGkhveI/LRMQS6FFM87gMzN4hIpsm1se9v4/tDlyZlBa/+yWWk1myn+w0n5zsUETlAuarR29cINMjU8s1y94S7ryQzyqwcwN2vdffx7j4FsOy+I4FRwDIzWwVEzWxZ8+uZWRHQHdjaFjfWkXh9kvg/F1P3ynJIpyGZ94n2Rdqzx4EbzawYdvbZuwZ4OK9R5VBs5hwC/UspOaPdtCSLyH5qcaJnZidlJ+/EzAaa2W+z6zG2ZGXrvY5Ay/or2dq5bBNtBbDCzIJm1jtbPg4YBzzl7o+6+4Amo9Zq3X1U9rVmAZ/PPv9P4JnO3j8vuaWGHbNeJ7nmXYqPHk7k44dpnVqRvbsCGEim60d3MjV5hwDfymdQuZJcu526R5YS/a8jsXB7avwRkf2xP3+9vwROyT7/WfbfOJmmz2l7O7GFI9CeBE42s0VkJgf9hrtvNbMS4IXsVB81wGez/e725i7g99kavnfJJJadlqfTxJ9bAkD0E+Mo6tstzxGJtG/Z2rs+ZLqH9CKT4K119015DSyHau96HdwpvUSDMEQ6sv1J9MrcfU22KfQUMm98DcCGlpy8rxFo2Rq3r2cfTY+pIzPydl+v36XZOWe3JK5Clq5LYOEgFggQ/dhYAqUlmh5BpAXc3c3sTaBrdgaALfs6p5B4MkXsztcpPmUURcM13ZJIR7Y/ffRqzKw/MAlY5O6NHZK1unU7lNz0HrG/vU793DUABHt1UZInsn/mkulC0unUPfo26fXvU3qpavNEOrr9+eS/lUxfuzDwtWzZccCS1g5KDpynnYb5a6l/YzWBrhFCw/rkOySRjupZ4Akz+w2Z6Zp29vN197vzFFNOxGZWERjUlZLTOmWeK1JQWpzoufv1ZvZ/QMrdl2eL15OZOV7agXRtA/Hnl5DauJ3QiL6UHDsKC6kWT+QAHUdm3rxJzcodKNhEL7lqG/VPLKPrdydhRRqwJdLR7VcW4O47F9Y2s5OAtLs/1+pRyQHxeAOprTFKji8nNKq/1qoVOQjuflK+Y8iH2K9fBzOiF0/Idygi0gr2Z3qV58zsuOzzbwH3AX8ys2+3VXCyb552EqvfASDYuwtdzzmacPkAJXkircDMeprZBWZ2dfbfgh6Z4IkUtXe9Tsknyika0j3f4YhIK9ifwRiHA69kn19CZimyY4DLWjsoaZn0jjpqH59P/JnFpKrfB1BTrUgrMbOPAMvJvMeNAy4FlmfLC1Ld35aQ3hyj9NLKfIciIq1kf7KCAOBmNhIwd18EmW+8bRKZ7FVizVbiLywFdyKTRhPs2zXfIYkUmpuAL2bXugXAzD4N3AIcnbeo2lBs5hyCQ7tTPHXUvg8WkQ5hfxK9fwG/IDNT/P8BZJO+d9ogLtmLujmraJi/lkDvLkQnH0qgWyTfIYkUogrggWZlDwK35yGWNpdctpX6v6+g6zUnYcFcrY4pIm1tf/6apwPvAfOB72fLDgVubt2QZF+CPaKExwyi9BMfUpIn0nbe5oOr6pxNpjm34MTumANBo/S/jsx3KCLSivZnepWtwLeblT3a6hHJbiVWVuPJFOHyAYRG9iM0sl++QxIpdF8DHjGzrwCrgWFAOXBaPoNqC16fpPaeNyiZNprgIC2RKFJI9mfUbcjM/tfMVphZXfbf/zWzcFsG2Nl5MkX8pbeJP7uExLItZFaKE5G25u4vASPJdFmZQ2bS+FHZ8oIS/7/FpN+ppfQyDcIQKTT700fvJ8BEMiPQVpNZ6/a7QDfgitYPTVLv1RJ/djHpbbWEjxhM8YRDNG2KSI6YWRlQ6+5/aFLW08wGuXuL1vjuKGK3VxEc0ZPij4/Idygi0sr2p4/e2cA0d3/K3d9y96eATwHntE1onVs63kDs4TfweILolMMoqRyOBdRBWiSH/goMblY2mOxgtEKRWFJNw3OrKb1kgt5jRArQ/tTo7akqSVVMrcjTjgWMQCRMyYdHUDS4J4Focb7DEumMKtz9zaYF7v6mmR2ar4DaQmzmHAgFiF6oQRgihWh/vr79GXjYzE4xszFmNpXMN94/t01onU/q3R3E/vY6yc3bAQhXDFCSJ5I/1Wa2y4Ry2e2teYqn1Xk8Qe1v3yDyqTEE+3fJdzgi0gb2p0bvm8D/ALcBg4D1ZJZBu6YN4upU3J3EW5uoe205Fg5llkwXkXy7G3jIzL4DrCAzMOOHwJ15jaoVxR9chG+r0yAMkQK210TPzD7arOjZ7MP4dzpyPPBMawfWWXhDkviLb5Nc9Q7Bsp5ETqggENFAZpF24DogAdwADAHWkEnyfp7PoFpT7PYqiip6E548LN+hiEgb2VeN3l17KG9M8hoTPg3VOkCJZZtJrn6H4sphhA8frFG1Iu3HJOBBd/+pmQ0Eriez5nc/YFNeI2sFiQWbaXhpLd1uOFnvOyIFbK+JnrsPz1UgnYm74zvqCHSNEBoziOCAHgR7leY7LBHZ1S+BU7LPf5b9NwHcAUzLS0StKDZzDhQHiX7+Q/kORUTa0P700ZNWkK5LUPevpaS21FD6qaMIRMJK8kTapzJ3X2NmRcBUYCjQAHT4OfTSsQZqfzePyH+OJdhH7z8ihUyTJuVQctP2zKja9dsoHj8UKwnlOyQR2bMaM+tPpgl3obvvyJa36A/XzKaa2VtmtszMrtrLcWeZmZtZZXY7ZGa/NbM3zWyxmV190HfSTPz+BXhNPaWXahCGSKFTjV4OuDsN89dSP3c11qWE0k98iGCfrvkOS0T27lZgNhAms+4twHHAkn2daGZBMjMUTAHWAbPNbJa7L2p2XFfgq8CrTYrPBord/QgziwKLzOxed191kPezU2zmHIrG9iV8/NDWekkRaadUo5cjqXdjFA3rS5dpRyrJE+kA3P164OPAce5+X7Z4PXBxC06fCCxz9xXu3kBmKqozdnPcNWQGedQ1vTRQmm0yjpBpLq45sLv4oIa5G0m8tp7SS4/SIAyRTkCJXhtKbthGanstZkbkxNFEJo3GwqpEFeko3H2puy9vtv3m3s7JKgPWNtlely3bycwmAEPc/dFm5z4IxICNZKZ0ucHd321+ATObYWZVZlZVXV3dshsCYjOroKSI6Oc0CEOkM1Ci1wY87dTNWUXtkwuon7sGAAsG9O1ZRAAwswBwI3DlbnZPBFJkJqYfDlxpZh+Ywsrd73D3Snev7Nu3b4uu6+6kN8eInnc4gZ6RA78BEekwVL3UytKxeuLPLSG1uYZQeX9KPjwy3yGJSO6tJzPJcqPB2bJGXcnMyfds9gvgAGCWmU0DPgM84e4JYIuZvQhUklmd46CYGb3/71w8mTrYlxKRDkI1eq0otTWzVm1q6w5KThxN5PgKLBTMd1giknuzgXIzG25mYeBcYFbjTnff7u593H2Yuw8DXgGmuXsVmebajwKYWSlwDC0YALI/rEjvSyKdhWr0WlGge4Sisp6Exw8l2D2a73BEJE/cPWlmlwNPAkHgbndfaGY/AKrcfdZeTr8NuMfMFpJZfeged5/f9lGLSCFSoneQ0u/XUTdnFZHjRmGhIiKTDs13SCLSDrj7Y8Bjzcq+t4djJzd5voPMFCsiIgdNid5BSKyqJv7i2wCkttVS1K9bniMSERER+TclegfAk2nqZq8gsWQjgT5diU4+lEDXknyHJSIiIrILJXoHoO7V5SSWbiJ8eBnFE4ZhQY1pERERkfZHid4BKP7QUIqG9iY0pFe+QxERERHZIyV6ByDQpZhAl+J8hyEiIiKyV2pzFBERESlQSvRERERECpQSPREREZECpURPREREpEAp0RMREREpUEr0RERERApUzhI9M5tqZm+Z2TIzu2oPx5xjZovMbKGZ/alJ+fVmtiD7+HST8rvMbJ6ZzTezB82sS7Z8uplVm9kb2cfFbX+HIiIiIu1LTubRM7MgcBswBVgHzDazWe6+qMkx5cDVwHHuvs3M+mXLPwFMAMYDxcCzZva4u9cAV2T/xcxuBC4Hrsu+5P3ufnku7k9ERESkPcpVjd5EYJm7r3D3BuA+4Ixmx1wC3Obu2wDcfUu2fCzwvLsn3T0GzAemZo9pTPIMiADe5nciIiIi0kHkKtErA9Y22V6XLWuqAqgwsxfN7BUzm5otnwdMNbOomfUBTgKGNJ5kZvcAm4BDgVubvN5ZTZp0h7AbZjbDzKrMrKq6uvqgblBERESkvWlPgzGKgHJgMnAe8Gsz6+HuTwGPAS8B9wIvA6nGk9z9QmAQsBho7L/3MDDM3ccBTwO/3d0F3f0Od69098q+ffu2yU2JiIiI5EuuEr31NKmFAwZny5paB8xy94S7rwSWkkn8cPdr3X28u08BLLtvJ3dPkWkOPiu7vdXd67O77wSOauX7EREREWn3cpXozQbKzWy4mYWBc4FZzY75K5naPLJNtBXACjMLmlnvbPk4YBzwlGWMypYbMA1Ykt0e2OR1p5Gp7RMRERHpVHIy6tbdk2Z2OfAkEATudveFZvYDoMrdZ2X3nWxmi8g0zX7D3beaWQnwQiaXowb4bPb1AsBvzawbmVq+ecAXspf8iplNA5LAu8D0XNyniIiISHti7hqoClBZWelVVVX5DkNEcsjM5rh7Zb7jOFh6/xLpfFr6/tWeBmOIiIiISCtSoiciIiJSoJToiYiIiBQoJXoiIiIiBUqJnoiIiEiBUqInIiIiUqCU6ImItAEzm2pmb5nZMjO7ai/HnWVmbmaV2e3zzeyNJo+0mY3PXeQiUkiU6ImItDIzCwK3AacCY4HzzGzsbo7rCnwVeLWxzN3/mF3ycTzwOWClu7+Rm8hFpNAo0RMRaX0TgWXuvsLdG8isxX3Gbo67BrgeqNvD65yXPVdE5IAo0RMRaX1lwNom2+uyZTuZ2QRgiLs/upfX+TRw7+52mNkMM6sys6rq6uqDjVdECpQSPRGRHMuu1X0jcOVejvkwUOvuC3a3393vcPdKd6/s27dvG0UqIh2dEj0Rkda3HhjSZHtwtqxRV+Bw4FkzWwUcA8xqHJCRdS57qM0TEWmponwHICJSgGYD5WY2nEyCdy7wmcad7r4d6NO4bWbPAv/t7lXZ7QBwDnBCDmMWkQKkGj0RkVbm7kngcuBJYDHwgLsvNLMfmNm0FrzEicBad1/RlnGKSOFTjZ6ISBtw98eAx5qVfW8Px05utv0smeZcEZGDoho9ERERkQKlRE9ERESkQCnRExERESlQSvRERERECpQSPREREZECpURPREREpEAp0RMREREpUEr0RERERAqUEj0RERGRAqVET0RERKRAKdETERERKVBK9EREREQKlBI9ERERkQKlRG8/eSLF9m8+Rf0zK/BEKt/hiIiIiOxRUb4D6GiSi6vZccur7PjpS1i3YkqmjqLk9AqKTy0n2Dua7/BEREREdlKit59C4wYwcOu3qH96OXUPL6Xu0aXEH1gIASN83BBKTh9NyekVFI3ug5nlO1wRERHpxJToHYBAaZjIJ8cQ+eQYPJ0mUbUhk/Q9/BY133yamm8+TXBUL0pOqyByegXhEw7BQsF8hy0iIiKdjBK9g2SBAOGJgwlPHEy3az5Kcs171D2ylLpHlhL71WxiN72CdW9s4h1NyamjCPRSE6+IiIi0PSV6raxoaA+6fHEiXb44kfSOeur/vuLfTbz3L4SgET5uKCWnV1Bympp4RUREpO0o0WtDgS7Fuzbxzt5A3cNvUffIUmq+8TQ138g28Z5eQeT00YSPH6omXhEREWk1SvRyxAIBwh8eTPjDg+n2w4/9u4n34aXEbptN7OfZJt5TyzO1fVPVxCsiIiIHR4lennygiffpFTv79sXvW7BrE+/powmN7pPvkEVERKSDUaLXDgS6FBP51Bgin9q1iTf+cJMm3vJeRE4fTclpFWriFRERkRZRotfOfKCJd3VjE+9b7PjFa+y48WWsR8nOiZpLTi0n0DOS77BFRApeTU0NW7ZsIZFI5DsUKXChUIh+/frRrVu3g34tJXrtXNEhPejypYl0+VKTJt6H36Lu0bf/3cR7/NCdEzWHKtTEKyLS2mpqati8eTNlZWVEIhHNliBtxt2Jx+OsX78e4KCTvZytdWtmU83sLTNbZmZX7eGYc8xskZktNLM/NSm/3swWZB+fblJ+l5nNM7P5ZvagmXXJlheb2f3Za71qZsPa+v5yobGJt+fdn2TAxivp+/JFdPnW8aS31VHz30+xZfQv2FRxC9uvfJL6Z1dqLV4RkVayZcsWysrKiEajSvKkTZkZ0WiUsrIytmzZctCvl5NEz8yCwG3AqcBY4DwzG9vsmHLgauA4dz8M+Fq2/BPABGA88GHgv82sMb29wt0/5O7jgDXA5dnyi4Bt7j4K+DlwfVveXz5YIED4mCF0v/Zj9J/3Bfqv+hrdf/EfFI3oyY5fvMY7J/2Wjf1+yrufeZDae98kvS2e75BFOpWWfLnNHneWmbmZVTYpG2dmL2e/9L5pZiW5iVr2JJFIEImom4zkTiQSaZVuArlqup0ILHP3FQBmdh9wBrCoyTGXALe5+zYAd29MY8cCz7t7Ekia2XxgKvCAu9dkX8+ACODZc84Avp99/iDwCzMzd2/cX3B2aeJ9v37XtXjvzTbxnnAIJadVqIlXpI01+XI7BVgHzDazWe6+qNlxXYGvAq82KSsC/gB8zt3nmVlvQJ3C2gHV5Ekutdb/t1w13ZYBa5tsr8uWNVUBVJjZi2b2iplNzZbPA6aaWdTM+gAnAUMaTzKze4BNwKHArc2vl00QtwO9W/eW2q9A12IiZ46l5z2fZMDG//53E+/W2p1NvJtH38r2/36S+udW4Uk18Yq0sp1fbt29AWj8ctvcNWRaHOqalJ0MzHf3eQDuvtXd9UcqIgckZ330WqAIKAcmA+cBvzazHu7+FPAY8BJwL/AysPNNz90vBAYBi4FPsx/MbIaZVZlZVXV1davcRHtjwSZNvPO/SP+VX6X7racSHN6DHbe+xjuTf8PGvmriFWll+/xya2YTgCHu/mizcysAN7Mnzex1M/vm7i7QGd6/pP1bsmQJZkZVVVW+Q5E9yFXT7Xqa1MIBg7NlTa0DXnX3BLDSzJaSSfxmu/u1wLUA2UEaS5ue6O6pbHPwN4F7mlxvXbYZpDuwtXlQ7n4H/7+9O4+PqjofP/55ZjIzCSEJZDcQDGvYDGgQUKDiTihgi7ggCtpSwYqUWr+AKCpuLD+FKEUtm0LFKtS9WrYqAhaRRZBCQMMeQCCRxUCSSTLn98cMQxYIBDKZMHner9e8mLnn3HvPGZKT557lXpgO0KFDh4Ad1i0pKKk+dYd1ou6wThUP8fZuQXDP5vosXqV8QEQswGTg/jMkBwFdgauBk8B/RGSdMeY/JTPVxvZLVd652u/LL7+cXbt2XfDxmzdvzoEDB4iO1ulANVV1BXprgOYi0hh3EHY3cE+ZPB/h7sl70zNE2wLY4ZnrUs8YkyMiKUAKsNgzL6+pMSbT874PsNVzrE+AQbh7//oBXwTy/LwLdWqIN6Rva0yxC+e3+9y3bvn0B47/ZTHH/7IYS0wd7F0b4eh2OfaujbC1j9ebNSt1bue6uA0D2gLLPH+I44FPRKQP7ove5caYbAAR+Rz3grRSgZ5S5+PAgQPe9//973+5/fbbWb9+PZdddhkAVuuZ23On04ndbj/n8a1WK/Hx8VVT2BrofL+Hmqxahm498+SGAYtwD7HON8ZsFpFnPQ0bnrQcEdkCfAn8nzEmB7ABKzzbpwP3eo4nwBwR2QRsAi4DnvUcaxYQJSKZwKPAWVe8KTexWnBck0jEizcRt8k9xFtvem8cac0p3HiQY48u4nDHGRyoP4Hsm+Zw/JkvyV+6HVdugb+LrlRN5L24FRE77ovbT04lGmOOGWOijTFJxpgk4BugjzFmLe628ArPvOQg4DpKL1xT6rzFx8d7X5GRkQDExMR4t8XExHjzjRs3jgcffJDIyEhuvvlmAF566SVSUlIIDQ0lISGBe++9t9QtP8oO3Z76/MEHH5CWlkadOnVo1qwZ77zzDhU5fPgw/fv3JzExkZCQEFq2bMnUqVPL5Xv77bdp3749wcHBREdH06tXL3JzcwH3/efS09Np2bIlDoeDuLg47rnndJ9SfHw8L730Uqnj3XvvvfTo0cP7uXPnzjz00EOMHj2a+Ph4WrRoAcCcOXO4+uqrCQ8PJyYmhj59+rB9+/ZSxzpw4AADBw4kNjaW4OBgWrZsydtvv01xcTENGzZk8uTJpfIfPXqUOnXqsGDBggq/m4tVbTdMNsZ8jnuuXcltT5V4b3AHZY+WyZOPe+Vt2eO5gC5nOVc+cMfFl7r2CkqqT9AfUgn9QyoAxfuOU/D1Hpwr3a9fnlsOLgNWwXblZe5ev66NsHdthDWurp9Lr5R/GWOKROTUxa0VmH3q4hZYa4z5pIJ9j4jIZNzBogE+P8M8PqWq3Msvv8zo0aNZvXo1xcXuqfAiQnp6Oo0bN2b//v38+c9/5r777mPRokUVHmvUqFFMnDiRqVOn8sYbb3D//fdz7bXXkpSUdMb8ea2tnjYAABw5SURBVHl5XHXVVYwcOZJ69erx1Vdf8dBDDxEdHU3//v0BeP311/nTn/7Es88+y2233UZhYSFLly71lnX06NG8/vrrTJo0iRtvvJFjx46xZMmSSn8Pb7/9Ng888ABffvml99hOp5Nx48aRnJzM0aNHefLJJ+nTpw8bN24kKCiI3NxcunXrRmRkJO+++y5JSUls27aN3NxcrFYrv//975k5cyaPPno6xJk3bx5hYWH85je/qXQZK0N0RNOtQ4cORieTnj/X8Xycq7JwrtxDwco9OL/JgvwiAKzNI71Bn6Pb5VibReo8P1Ujeea+dTh3zppN2y/fy8jIoFWrVqW2HR3xbwo3/FTtZbG1j6deelql91u2bBnXX389e/fupWHDhqXS4uPjSU1N5bPPKr6mWLVqFddeey3Z2dlERUWxdetWWrVqxZo1a+jQoYP387Rp0/jjH/8IuIOksLAwpk+fzqBBg867vEOGDGH//v18+umnGGOIj4/nvvvuK9crB3DkyBHi4uKYPHkyw4YNO8PR3HV87LHHeOyxx7zb7r33XrKzs1m4cCHg7tHLzc1l06ZNFf7dOnDgAAkJCaxdu5bU1FSmTZvGqFGj2L59O3FxceXy7927l8aNG7Ns2TK6du0KwJVXXsktt9zCxIlnv9XvmX7uTjnf9ksfgaYuiCU8mOBbmxF8azMAjLOIwvUH3EHfij3kfbyNk29ucOeNDT3d49fNM88vSOf5KaVUTdKxY8dy25YuXcrEiRPZunUrR48exeVyAbB7926ios5+17L27dt739vtdqKjozl48OBZ8xcVFTFhwgTmz5/Pvn37KCgowOl00rJlS8AdKB06dIhbbrnljPtv2rSJwsLCs6ZXxtVXX10uyFu3bh3PPvssGzduJCcnh1OdZLt37yY1NZV169aRkpJyxiAPIDExkbS0NGbMmEHXrl1Zu3YtGzduZP78+Rdd3nPRQE9VCbEHYe+ciL1zIjzWBeNyUbQtx93jt2I3zpV7yP8gw5031Ia9c0PsngUe9k4NsNR1+LkGSilVORfSq1aThYaGlvqcmZlJr169GDx4MOPGjSMqKort27fz61//GqfTWeGxyi5gEBFvkHgm48ePZ8qUKUyZMoWUlBTCwsKYMGECK1euvPAKlWGxWCg7inmmJ0+U/R6OHTvGzTffzM0338ycOXOIi4vD6XTSrl27c34PJQ0dOpQ77riDV155hZkzZ9K9e3eaN29+YZWpBA30lE+IxYKtVQy2VjHl5/mt8MzzG7fMPQPp1Dy/bp5evy46z08ppfxt9erVFBYWkp6eTlCQO1z4+uuvfXKu5cuX07t371JDuz/8cPpOaomJicTGxrJ48eIz9tpdccUV2Gw2Fi9e7F1AUVZsbCz79+/3fjbGsGHDBho3blxh2f73v/9x5MgRJkyY4M37xRdflMqTmprK/PnzOXjw4Fl79dLS0oiJiWH69On84x//4I033qjwvFVFAz1VbawNwqlzZ1vq3NkWANexfJzfZOFcsZuClXs48fpaTkz5xp23eaT3li6Oro10np9SSlWzFi1a4HK5mDJlCv369WP9+vWMHz/eJ+dKTk7mww8/ZMWKFcTGxjJr1iw2btzovQ2MiDB27FgeffRRoqOj+c1vfkNRURH/+c9/uP/++6lfvz7Dhw9nzJgx2Gw2brjhBk6cOMGiRYsYNWoUADfddBNz5syhd+/eJCQkMHXqVH766adzBnqNGzfGZrPx6quv8sgjj5CZmcnjjz9eKs/AgQN5+eWX6d27tzcgzMzM5NixY/Tr1w9w9ygOHjyYsWPHEhYWRt++fX3wTZZXk56MoWoZS4R7nl/48zcSs+wBEo6NJvq/vyd80s3YWsWQ99FWjv7uYw62mMpPl71ETr/3yE1fhXPtPn1sm1JK+djVV1/N5MmTeeWVV2jdujVTp05lypQpPjnXuHHj6NSpEz179qRLly44nU6GDh1aKs+wYcOYPn068+bNIyUlhe7du7N06VLvvQAnTZrEU089xUsvvUSbNm3o0aMHmzZt8u7/5JNPctNNN9G3b1+6d+9OQkICvXv3PmfZEhISmDNnDp988gmtW7dmzJgx5b6HsLAwVqxYQbNmzbjjjjto1aoVw4cPp6Cg9C3IBg8eTFFREYMGDcLhqJ4pS7rq1kNXrdU8xuWiaGv26ZW9K/ZQvOso4Jnnd02ie45f10bYOzfEEnpp39RSVT9ddavOV0WrH5U6X+vXryc1NZWMjAzvQpOK6KpbFdDEYsHWOhZb61hCH3T/LBdnHaPg673u+/mt2F16nt9Vl51+ikeXRKyxOs9PKaWU/+Xn55Odnc3jjz9OWlraeQV5VUUDPXVJsTaMoM5dEdS5q8Q8v1V7Pat793DitTXeeX5BLaLcvX2eRR7WpjrPTymlVPV76623ePjhh2nbti0ffvhhtZ5bAz11SbNEBBPcoznBPdxL1E1BEc71B3B6bumS92EGJ2d/584bF1r6ub3t4vR+fkoppXxu6NCh5eYcVhcN9FRAEUcQjmsScVyTCCM98/wySszzW7mH/PdL3M/vmkTsnRpgax+PrX081ib1EYuuUVJKKRUYNNBTAU0sFmxtYrG1iSV0SIl5fitLPLd3wkoodi9Kkrp2bO3ivIGfrX08traxSLDNn9VQSimlLogGeqrWsTaMoM7dV1Dn7isAMPmFFG4+TOGGn7yvk3M3Yqat8ewgBLWMLh38tY/HGh1awVmUUkop/9NAT9V6EmzDnpqAPTXBu824XBTvPFoq+HN+tZu8eafvyWRpEIatfTz2ksGfDv0qpZSqQTTQU+oMxGIhqGkkQU0jCbm9tXd7cfYJCjceLBUA/rIws+Kh3zaxSIgO/SqllKp+GugpVQnW6FCsNzYh+MYm3m0XNPTbLh5rjA79KqWU8i0N9JS6SOc19LvxJ5zLdehXKaVU9dJATykfOOvQb85Jijb+hLOiod+UM6z61aFfpZRSF0ADPaWqkTWqDtYbmuC4oczQ75YyQ79/34h5zTP0aznLql8d+lVKnYf777+fOXPmABAUFERERAQtW7akT58+PPzww4SGalsSyDTQU8rPJNiG/aoE7FeVGfrdVWbV74rd5L1TYug3Icwb9J0a/rU21aFfpVR53bp1Y/78+bhcLnJycli5ciXjx49n1qxZLF++nLi4OH8X0S+cTid2u93fxfAp/YugVA0kFgtBTSIJ6dua8GdvIOqTe4jf8yjx2SOJ/s9Awl++BceNjSnee4zciSv5+c4FHGwxlQPh4zncZRZHH/6MEzPW4VyzD5NX6O/qKKX8zG63Ex8fT0JCAldccQUPPfQQq1at4vDhw4wePbpU3qlTp9KyZUuCg4Np3rw5L7zwAkVFRQA88cQTJCcnlzv+Qw89RNeuXc96/iVLltC9e3ciIyOJiIjguuuu49tvvy2VJzc3lxEjRpCYmIjD4SApKYkXX3zRm37o0CEeeOAB4uLiCA4OJjk5mdmzZwOwbNkyRISsrKxSxwwKCuKtt94CYNeuXYgI8+bNo2fPnoSGhjJ27FiMMfzhD3+gadOmhISE0KRJE8aMGUNBQUGpYy1dupRu3bpRp04dbx22b9/OsmXLsFqt7N27t1T+uXPnEhERwYkTJ876vVQH7dFT6hJywUO/yVEEtYwmqEWU+9Xc/a8lNhQR8VNtlLr0nfj39+W22ZKisbdKwBQVc3LJ5vLpzeKwN4/DlV9I3pcZ5dLtyZdhaxKDK7eAvBXbyqWHpqVUSdkbNGjAgAEDmDt3LrNmzcJisfDMM8/w5ptvkp6eTvv27cnIyGDo0KHk5+fz3HPPMWjQIF588UVWr15Np06dACgoKOC9995jwoQJZz1Xbm4uf/zjH2nXrh1FRUVMmTKFHj168OOPPxIVFYUxhl69erFnzx6mTp1KSkoKWVlZbNvmrn9eXh7XXXcdISEhzJs3jyZNmpCZmcnPP/9c6XqPGjWKiRMnMm3aNACMMcTGxvLOO+8QFxfH999/z5AhQ7DZbIwbNw5wB3m33norjzzyCH/9619xOBx8/fXXFBYW0r17d5o3b87s2bN5+umnveeZMWMG99xzj9+HxjXQU+oSd15DvxsPUpSRTf6/foBC1+l9wx0lgr/IUoGgJSLYH9VRSlWjNm3acPz4cbKzs6lbty6TJk3igw8+oEePHgA0btyY559/nuHDh/Pcc8/RokULOnXqxNy5c72B3qeffkpeXh533nnnWc/z29/+ttTn6dOn8/7777Nw4UIGDBjAF198wVdffcWaNWvo0MH9uMomTZrwq1/9CoB33nmHnTt3kpmZScOGDb3pF2LIkCEMGDCg1LYXXnjB+z4pKYnt27fz2muveQO9cePGkZaWRnp6ujdfy5Ytve8ffPBBXnnlFcaOHYvFYmHr1q2sXLmSV1999YLKWJU00FMqAJ0a+j01/HuKKSqmeM8xin7IOf368Wec/91L3j82gTl9DEts6OnAr2Qw2DRSVwEr5VFR75oEWStMtwTbKk6v66iy3ruzMcaz4l+EzZs3k5eXx+23316qp7+4uJj8/HwOHz5MTEwMgwYNYuzYsaSnp2Oz2Zg7dy59+vShXr16Zz3Pzp07eeqpp1i1ahWHDh3C5XJx8uRJdu/eDcC6deuoX7++N8gra926dbRu3dob5F2Mjh07lts2Y8YMZs6cya5duzhx4gRFRUW4XKcvitetW1dhj+WgQYN44oknWLRoEWlpacycOZPU1FSuvPLKiy7vxdJAT6laRIKs3gCQHs1LpZn8Qoq2H6Hox5xSgWD+Zz/gml1ijomAtVFEqSHgUy/r5RFIkLWaa6WUulCbN28mIiKCqKgoduzYAcCCBQto0aJFubyRkZEA3H333YwYMYLPPvuMLl26sHDhQj766KMKz9OrVy+io6OZNm0aiYmJ2O12unbtitPprJJ6WDyL0E4FruAOUEsGa6eUHUpdsGABDz/8MBMmTOC6664jPDycBQsW8MQTT5z3+aOioujXrx8zZszgxhtvZO7cuTz//PMXWJuqpYGeUgpwDwHb2sRiaxNbLs11PJ+iH38u0Qvo/vfk299jjpeYsGyzENSkfpleQM98wIQwnQ+oVA2yb98+5s2bR9++fbFYLLRp04bg4GB27NhBz549z7pf/fr16d27N3//+9/Zs2cPkZGR3HrrrWfNn5OTw5YtW/j888+9+bKysjh06JA3T2pqKkeOHGHt2rVn7NVLTU1l9uzZZGVlnbFXLzbW3W7t37+fxMREADZs2FAq8Dub5cuXc+WVV/Loo496t+3atavc+RcvXszw4cPPepwhQ4Zw/fXX87e//Y28vDz69+9/znNXBw30lFLnZAkPLvf0D3BfPbsOn/AOAZfqCVyyA/KLvHkl1OYO+krOBfS8LJF1qrtKStUqTqeTn376qdztVWJjYxk/fjwAdevWZcyYMYwZMwYR4aabbqKoqIhNmzbx3XffMXHiRO/xBg4cyB133EFGRgYDBgzAaj17T379+vWJiYlhxowZNG3alJycHEaOHElISIg3zw033EC3bt246667mDx5MikpKezfv5+MjAwGDx5M//79mTRpEn369GHSpEk0bdqUHTt2kJ2dzV133UWzZs24/PLLeeaZZ5gyZQrZ2dneepxLcnIys2bN4uOPP6Zt27b861//4oMPPiiVZ+zYsaSlpTFixAh+97vf4XA4WLVqFddcc413FXLXrl1JTk7mscceY+DAgYSFhVXq/8hnjDH6MobU1FSjlKo6ruJiU7j7iMlbkml+mbbaHBnxb3O459vmQLNXTJb1GZPF097XvsgJ5mDnGSZn4Afm+PNfmRPvbTIF3+03xbkFPi0jsNb4qE0BegDbgExgdAX5bsc9O7KD53MSkAds8LzeONe5tP3yvS1btvi7CBds0KBBxvMzZqxWq4mMjDRdunQxEydONLm5ueXyz5gxw7Rr1844HA5Tr14907FjR/Paa6+VyuN0Ok1MTIwBzIYNG85ZhmXLlpmUlBTjcDhMixYtzD//+U/TtGlT8/TTT3vzHD9+3AwbNszEx8cbm81mkpKSzPjx473pBw4cMPfdd5+JiooyDofDJCcnmzfffNOb/s0335irrrrKBAcHm5SUFLN8+XJjtVq9eXbu3GkAs2LFinJ1efDBB039+vVNWFiY6d+/v5k6dapxh0inLVy40HTu3NkEBweb8PBw0717d7N9+/ZSedLT0w1gvv3223N+J+ejop+7822/xJxHt2Zt0KFDB7N27Vp/F0OpWsE4iyjaebTcfMDiH3+mOOt4qbyWBmHl5gIGtYgiqHE9xH5xgxIiss4Yc+bZ3xd3XCvwA3AzkAWsAfobY7aUyRcGfAbYgWHGmLUikgT8yxjT9nzPp+2X72VkZNCqVSt/F0PVcCNHjmTJkiV89913VXK8in7uzrf90qFbpVS1E3sQtuRobMnR5dJcJ5wUZ5afD5j//hZcOXmnM1oFa1K9cnMBHTc1qQlzATsCmcaYHQAi8i5wG7ClTL7ngInA/1Vn4fxx7zd7mwbYGkVRfOwk+f/NLJfuaJdIUEJ9inNyyf92R/n0q5IIigun6OBxCtbvKpce3LEJ1qi6FO0/QsHGveXTr22GNaIOhXtycG7eVy49pFsylroOCnccxrntQLl0c7n7z6UrvxBTUP4m5JbwEEQEV54T4ywql26NcE9PcJ10YgrLpgvWiBBPegGmsLhMsmAN96SfKMAUlU4Xi2AJ86Tn5mOKSy9AEKsFS93g80v/JQ/jKt0BJEFWLKEOAIqP50GZDiKxWbHU8aQfy6PU8n1AbEFY6tg96ScpS+xBWELs7qkgx/PKpztsWIJtGJcL1y/55dODbVgcNkyxC1fumdLtWBxBmKJiXCcKyqVbQuyIvYL0Og7EZsUUFuM6eYb0UAfHT+SybfMWpv9tOukT/1+pelpCHUiQFVdBESa//OITS91gxGrBVVCIxVH1dzTQQE8pVaNYQu1Y2sVjaxdfLs3188lycwGLfszh5PLdmBOFSL1gLvt5lB9KXU4DoGS0kQV0KplBRK4CEo0xn4lI2UCvsYh8BxwHnjTGrCh7AhF5EHgQoFGjRlVZdqVUJd12222sXr2au/r2Y8BdNWMRxik6dOuhQx9KXbqMMbgO/ELx/l+wd2hw3vv5cOi2H9DDGDPY8/k+oJMxZpjnswX4ArjfGLNLRJYBj3mGbh1AXWNMjoikAh8BbYwxx894MrT9qg46dKv8oSqGbvVZt0qpS56IYE0Ir1SQ52P7gMQSnxt6tp0SBrQFlonILqAz8ImIdDDGFBhjcgCMMeuA7UD5m5oppdR50EBPKaWq3hqguYg0FhE7cDfwyalEY8wxY0y0MSbJGJMEfAP08fToxXgWcyAiTYDmQPlJa6ra6QiYqk5V9fOmgZ5SSlUxY0wRMAxYBGQA840xm0XkWRHpc47dfwV8LyIbgH8CQ40xlX9yu6pSNpuNvLzyCwWU8pW8vDxstotfnKGLMZRSygeMMZ8Dn5fZ9tRZ8nYv8f594H2fFk5VWmxsLPv27aNBgwaEhITUhJXdKkAZY8jLy2Pfvn3ExcVd9PE00FNKKaXOITw8HHA/YquwsPztVZSqSjabjbi4OO/P3cXQQE8ppZQ6D+Hh4VXyh1ep6qRz9JRSSimlApQGekoppZRSAUoDPaWUUkqpAKWBnlJKKaVUgNJATymllFIqQOmzbj1E5DCwuxK7RAPZPipOTVEb6ghaz0BS2TpeboyJ8VVhqou2X2dVG+pZG+oIWs8zOa/2SwO9CyQia33xMPSapDbUEbSegaQ21LEq1JbvqTbUszbUEbSeF0OHbpVSSimlApQGekoppZRSAUoDvQs33d8FqAa1oY6g9QwktaGOVaG2fE+1oZ61oY6g9bxgOkdPKaWUUipAaY+eUkoppVSA0kCvkkSkh4hsE5FMERnt7/L4gojMFpFDIvI/f5fFV0QkUUS+FJEtIrJZRP7k7zL5gogEi8i3IrLRU89x/i6Tr4iIVUS+E5F/+bssNVVtaL9A27BAom3YxdNArxJExApMA9KA1kB/EWnt31L5xFtAD38XwseKgL8YY1oDnYGHA/T/sgC4wRjTDmgP9BCRzn4uk6/8CcjwdyFqqlrUfoG2YYFE27CLpIFe5XQEMo0xO4wxTuBd4DY/l6nKGWOWAz/7uxy+ZIw5YIxZ73n/C+5frgb+LVXVM265no82zyvgJuaKSEPg18BMf5elBqsV7RdoGxZItA27eBroVU4DYG+Jz1kE4C9WbSMiScCVwGr/lsQ3PMMBG4BDwBJjTCDWMx0YCbj8XZAaTNuvAKVtWEDwWRumgZ6q1USkLvA+MMIYc9zf5fEFY0yxMaY90BDoKCJt/V2mqiQivYBDxph1/i6LUtVN27BLn6/bMA30KmcfkFjic0PPNnUJEhEb7gZynjHmA3+Xx9eMMUeBLwm8uUtdgD4isgv3cOQNIvK2f4tUI2n7FWC0DQsYPm3DNNCrnDVAcxFpLCJ24G7gEz+XSV0AERFgFpBhjJns7/L4iojEiEg9z/sQ4GZgq39LVbWMMY8bYxoaY5Jw/05+YYy518/Fqom0/Qog2oYFDl+3YRroVYIxpggYBizCPfF1vjFms39LVfVE5B/AKiBZRLJE5Pf+LpMPdAHuw33ltMHz6unvQvnAZcCXIvI97j/0S4wxevuRWqi2tF+gbViA0TbsIumTMZRSSimlApT26CmllFJKBSgN9JRSSimlApQGekoppZRSAUoDPaWUUkqpAKWBnlJKKaVUgNJAT6nzJCJJImJEJMjfZVFKqcrSNqx20kBPKaWUUipAaaCnlFJKKRWgNNBTlzQRSRCR90XksIjsFJHhnu3PiMg/ReQ9EflFRNaLSLsS+7USkWUiclRENotInxJpISLysojsFpFjIrLS8+idUwaIyB4RyRaRJ6qxukqpAKNtmPI1DfTUJUtELMCnwEagAXAjMEJEbvVkuQ1YAEQC7wAfiYjN8yDwT4HFQCzwCDBPRJI9+70EpALXevYdCbhKnLorkOw531Mi0spnlVRKBSxtw1R10EegqUuWiHQCFhhjGpXY9jjQAtgN9DDGdPZstwD7gDs9WRcACcYYlyf9H8A24FngBNDZGLOxzPmSgJ1AojEmy7PtW2CyMeZdH1VTKRWgtA1T1UFX3qhL2eVAgogcLbHNCqzA3UjuPbXRGOMSkSwgwbNp76kG0mM37ivqaCAY2F7BeX8q8f4kUPeCa6CUqs20DVM+p0O36lK2F9hpjKlX4hVmjOnpSU88ldFzNdwQ2O95JXq2ndII99VyNpAPNK2WGiilajNtw5TPaaCnLmXfAr+IyCjP5GOriLQVkas96aki0tdzz6gRQAHwDbAa91XsSM98l+5Ab+BdzxXybGCyZ5K0VUSuERFHtddOKRXotA1TPqeBnrpkGWOKgV5Ae9zzTrKBmUCEJ8vHwF3AEeA+oK8xptAY48TdKKZ59nkNGGiM2erZ7zFgE7AG+BmYiP6uKKWqmLZhqjroYgwVkETkGaCZMeZef5dFKaUqS9swVVU0wldKKaWUClAa6CmllFJKBSgdulVKKaWUClDao6eUUkopFaA00FNKKaWUClAa6CmllFJKBSgN9JRSSimlApQGekoppZRSAUoDPaWUUkqpAPX/AX+W0ggv8E7+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(runner, \"fw-zero.pdf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从输出结果看，二分类准确率为50%左右，说明模型没有学到任何内容。训练和验证loss几乎没有怎么下降。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "为了避免对称权重现象，可以使用高斯分布或均匀分布初始化神经网络的参数。\n",
    "\n",
    "高斯分布和均匀分布采样的实现和可视化代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHW5JREFUeJzt3Xu4HXV97/H3h2D0UVDQRMQkEJDYNlUP0l3AS5FSbMPlBH3QnqC03DSlmsopHBUK5WDU4wHOgWpNqxRBq0K42B6jxAdv0Io1NBvlYohAiFwSEDbhfhEIfM8fv9k4WVlrr1l7z1qz1qzP63nyZM3Mb89811y+6ze/+c2MIgIzM6uXbaoOwMzMyufkbmZWQ07uZmY15ORuZlZDTu5mZjXk5G5mVkNO7h2QtIukxyVNq2j5fyPp/BLn97ik3bPPX5b0qRLn/QVJf1vW/Ky4xnUv6S8l3Zdt71dVGVsrktZI2r+keb1f0ndzwyFpjzLmnc3vheOmn2lQ+7lLWgT8NfAG4Angl8BXgH+MAfxSkq4G9gWeBQK4DbgMODcinp7EvL4WEYV/CCR9GdgQEad1sqzsb48GPhARb+/0b21rkgKYFxHrcuPOAPaIiCM7nNeLgEeBfSPihlIDLbb8uaRj84ls1BPAauCzEfG9Sc7rRRGxuYO/22p9dvC3V9PhsdQvBrLmLukk4LPA2cBrgJ2A44G3AdMrDG2qlkTE9sDOwEnAImClJJW5EEnbljk/62s7AS8B1nT6h0rKyhE7RMR2wH8Bvgf8a1YpKJX37ZyIGKh/wCtIv/6Htyl3CPAzUq3lbuCM3LT9SbXUfPk7gAOzz3sDo9nf3geck42fS6pVb5sNHwOsBR4D1gN/0bgMUpK+H7gXOGaCeK8m1X7z43YBngQOzYbPINUiIB2wXwM2AQ+TakM7AZ8GngN+DTwOfD4rH8CHSWcEv8yN2yP7/GXgC6QD7zHg34Bdm33vfLzA72TLei5b3sO5+X0qV/6DwDrgQWAF8NrctCD9ON+WfZdlZGeVw/gvv11y4/LbfsJ9a3zdA68nHSuRbZsfZtPfmu0vj2T/v7Vhu34a+DHwFLBHNu5TwH9k8/kW8Crg66RjZDUwt8V32Wrfycb/D9KxtU0Hx99due/yOPAW4Ogs1nNJx8KnsnHXNKzPj5CO0QdIlcJtGtdrY7xMfCyNHzevAP4ZGAPuBE7Lzfto4Brg/wAPkc46DurVfjSINfe3AC8Gvtmm3BPAnwM7kBL9X0p6V8FlfJZ02vhy4HXApS3K3Q8cCryclOjPlbRXbvprSBt/FnAcsEzSjgVjICLuIu3kf9Bk8lHZvOeQDrTjgaci4lTgR6SzgO0iYknub94F7APMb7HI9wOfBGYA15MO3nYxrs2W/ZNseTs0lpF0APAZ4E9JZyV3Assbih0K/D7wpqzcn7Rb9pBru29FxK3A72aDO0TEAZJeCVwBfI6035wDXNHQFv9nwGJge9K2gnQW+WfZ8l4H/AS4EHglqYLzPzuM/1+AVwO/1WRaq+Nvv9x32S4ifpIN70NK3OOVm2beDYwAewGHAce2C7DNsTTu70nbYXfgHaScc0xu+j7ALaRj6izgS2WfibcyiMl9BvBA5NrcJP2HpIclPSVpP4CIuDoiboqI5yPiRuBi0sov4llgD0kzIuLxiFjVrFBEXBERt0fyb8B32TIRPwssjYhnI2Il6de/2c48kXtIB1CzGF9FqkE8FxHXRcSjbeb1mYh4MCKeajH9ioj490ht/KcCb5E0p8N4m3k/cEFE/DSb9ynZvOfmyvzviHg4+0G7CtizhOXW2WT3rUOA2yLiqxGxOSIuBn4B/NdcmS9HxJps+rPZuAuzff0R4DvA7RHx/ew4vAx4c4fx35P932rfbnv85ecVEX+fxdtq3z4z2/fvAv4OOKLDeLeSdaxYBJwSEY9FxB3A/yX9CI67MyL+KSKeI10T3Jn0I9R1g5jcNwEz8m1rEfHWrMa4iew7SdpH0lWSxiQ9Qqpdzii4jONIp7S/kLRa0qHNCkk6SNIqSQ9Kehg4uGEZm2LLCz9PAtsVjGHcLFJTRqOvAlcCyyXdI+ms7OLZRO4uOj0iHs+W+9pOgm3htfymBjg+702k7zbuV7nPk1lPdfIc0LgtX0RKeuMmu29tsS0yd7Lltmi2n9yX+/xUk+HJ7NfQfN8udPzltNuvG8vcSTn79QzSdsmvz8Z1+cJ+HRFPZh97sm8PYnL/CfA06dRqIheR2nbnRMQrSO3J46dDTwAvHS+Y/QLPHB+OiNsi4gjSaeOZwOWSXpafuaQXA98gtaftlP24rMwtY8qyWvPvkU4Nt5DV2D4REfNJbaiHkk4JIbUJNtOuF9ELtXRJ25FqVffwm54OL82VfU0H870H2DU375eRzjo2tvm7YXUXqe03bze2TsqTscW2yOzCltuiF73N3k1q1rylccIEx99k92vI7duk7zt+5rBFLmDL/brdvB8g/eDm12fjuqzMwCX3iHgY+ATwD5LeI2l7SdtI2hPIJ+DtgQcj4teS9gbel5t2K/ASSYdktd3TSO34AEg6UtLMiHiedIEP4PmGUKZnfzMGbJZ0EPDHZXxHSS+V9A7SdYX/JP1oNJb5Q0lvzH6YHiXtZOMx3kdqA+zUwZLeLmk6qe19VUTcHRFjpB32SEnTJB1Lagsddx8wO/u7Zi4GjpG0Z/aj+L+Aa7PTWNvaJcBpkmZn+/aBpGaTy0uY90rg9ZLeJ2lbSf+NdA3m2yXMuy1JO0laQmqjPyU7xhrLtDr+xrL/J7Nvf1TSjlmF6QTSOoZ0bWm/7B6WV5CaDPNaHktZU8ulwKezPLQrcCKpo0PlBi65A0TEWaSV+DHSyr8P+CLwcdIVfYAPAUslPQacTu6iaNZu+CHgfFLSeoLU+2DcAmCNpMdJF3cWNbblRcRjpCvwl5KuhL+PdKYwFZ/P4r2P1C74DWBBswOAVMO4nJTY15J6t3w1m/ZZ4D2SHpL0uQ6WfxHpoHuQdMaQ71P9QeCjpOaU3+U36xngh6Sudr+S9EDjTCPi+8DfZt/nXtIPw6IO4ho2S0nr9xrSvnUW8P6I+PlUZxwRm0hneSeRtuXHSL2xttpuJXtY0hPATaTmy/dGxAUtyjY9/rJmjU8DP86use3bwfK/CVxHSuZXAF8CiNTX/hLgxmx6449cu2Ppr0j5Yz1pe10EtPpePTWwNzGZmVlrA1lzNzOziTm5m5nVkJO7mVkNObmbmdVQZQ/ZmTFjRsydO7eqxVvNXXfddQ9ExMz2Jcvnfdu6qei+XVlynzt3LqOjo1Ut3mpOUhk3/EyK923rpqL7tptlbGhJWiDpFknrJJ3cZPq5kq7P/t2aPWLCbCD42cc2lLI7e5cB7yTdwLZa0oqIuHm8TET8da78X9H5w7HMKuOauw2rvYF1EbE+Ip4hPYJ4oucVHUF6jILZQHByt2E1iy2fFLiBLZ/m94LsmSG7kR6z0JSkxZJGJY2OjY2VGqjZZDi5m7W3CLg8e1BUUxFxXkSMRMTIzJmVdNIx24KTuw2rjWz5GNjZtH5U6yLcJGMDxsndhtVqYJ6k3bJHFS+iyVM9Jf02sCPpPQJmA8PJ3YZS9hajJaS3Wa0FLo2INZKWSlqYK7oIWB5+fKoNGHeFtKGVvXt0ZcO40xuGz+hlTGZlcc19wDx64VZv3DOr1EadUXUI1oSTu5lZDTm59zHX0s1sspzczaxjborpf07uZmY15ORuZlZDTu5mZjXk5G5mVkNO7n3GPWTMrAxO7mZ9yL1RbKqc3AeYa/lm1oqTu5lZDTm5mw0JN/UMFyd3M7MaKpTcJS2QdIukdZJOnqDc4ZJC0kh5IZqZWafaJndJ04BlwEHAfOAISfOblNseOAG4tuwgh1n+oqkvoBq4ecWKKVJz3xtYFxHrI+IZYDlwWJNynwTOBH5dYnxmZjYJRZL7LODu3PCGbNwLJO0FzImIKyaakaTFkkYljY6NjXUc7LBwDd3MpmrKF1QlbQOcA5zUrmxEnBcRIxExMnPmzKkueqj5B8Amo+5NOnX/fp0oktw3AnNyw7OzceO2B94AXC3pDmBfYIUvqpqZVadIcl8NzJO0m6TppLfBrxifGBGPRMSMiJgbEXOBVcDCiBjtSsRmZtZW2+QeEZuBJcCVwFrg0ohYI2mppIXdDtCsW4p08ZX0p5JulrRG0kW9jtFssrYtUigiVgIrG8ad3qLs/lMPy6y7cl1830nqJLBa0oqIuDlXZh5wCvC2iHhI0quridasc75DtU9M9QKpL7B2rEgX3w8CyyLiIYCIuL/HMZpNmpP7APKNTaVo28UXeD3wekk/lrRK0oJWM6tLN9869zbp9LsN+rpwcjdrbVtgHrA/cATwT5J2aFbQ3Xyt3zi527Bq18UXUm1+RUQ8GxG/BG4lJXuzvufkbsNqwi6+mf9HqrUjaQapmWZ9L4McVIPepFEHTu42lAp28b0S2CTpZuAq4KMRsamaiM06U6grpFkdteviGxEBnJj9MxsorrkPAPeIsapU0bwybE06+e9b5nd3cu8jTuJmVhYndzOzGnJyN6tAHZse6vidGvXiO5a1DCd3M7MacnLvA36ujJmVzcndzJqaTPNAt5stNuqMgW7+6WXsTu59yrVxM5sKJ3czsxpych8QrsnXQ+NpebdO07t5+j+VeU/m+4+XGeTmmCo4uZuZ1ZCTu5lZDTm514ybb+qrVbNEq2eTtGvGmEzPkyLNKp00tRQt3+359aKXT685uZuZ1ZCTu5lZDTm5m/WxfnmWSZEmnqqV8T2qVHZsTu5mZjXk5G5mVkNO7mYDqkjvmX7Sy5uRutmMNCg3VTm5m5nVkJN7jbiPu5mNc3K3oSVpgaRbJK2TdHKT6UdLGpN0ffbvA2Ust2iTQS9P+/u9iaFbyu5h00/rcduqAzCrgqRpwDLgncAGYLWkFRFxc0PRSyJiSc8DNJsi19xtWO0NrIuI9RHxDLAcOKzimMxK4+Ruw2oWcHdueEM2rtHhkm6UdLmkOb0JzWzqnNwr5ougfe1bwNyIeBPwPeArrQpKWixpVNLo2NhYzwLMK7O9t6x5lfns924soxv65cFlTu42rDYC+Zr47GzcCyJiU0Q8nQ2eD/xeq5lFxHkRMRIRIzNnziw9WLNOObnXlM8I2loNzJO0m6TpwCJgRb6ApJ1zgwuBtT2Mz2xKnNxtKEXEZmAJcCUpaV8aEWskLZW0MCv2EUlrJN0AfAQ4uopYJ/vM9Drrp9cI9uv2cVdIG1oRsRJY2TDu9NznU4BTeh2XWRlcczczqyEnd7Mh0w89YfrBRPHXYR0VSu4FbtM+XtJN2S3a10iaX36oZmZWVNvknrtN+yBgPnBEk+R9UUS8MSL2BM4Czik9UivMPWXMrEjNve1t2hHxaG7wZUCUF6KZTfb0vswbanpxQ1KZf79RZ5TS82VQFekt0+w27X0aC0n6MHAiMB04oNmMJC0GFgPssssuncZqZmYFlXZBNSKWRcTrgI8Dp7Uo47v4Mr1oOnHzjNnwKpLc296m3WA58K6pBGVm5aqiuWEYmkSK9Lip6ianIsm9yG3a83KDhwC3lReimZl1qm2be0RsljR+m/Y04ILx27SB0YhYASyRdCDwLPAQcFQ3gzYzs4kVevxAgdu0Tyg5LjMroIpX8vWToq/JmxWty1XVfNIurqnyHapDwhdXzYaLk7uZWQ05uVfAtWiDLU/569ysUufvNq6XN5kV5eRuZlZDTu5DwGcKZsPHyd3MrIBBa15ycu8x16LNrBec3GvOPyZmw8nJ3axPdOu0v+h8+6HZoZMY+iHeyehV3E7uZmY15ORuZlZDTu421Nq9HzhX7nBJIWmkl/E1GoRH91Y936q0+z69/r5O7ja0Cr4fGEnbAycA1/Y2QrPJc3KvkHuyVK7t+4EznwTOBH7dy+DMpsLJfYj4x2Qrzd4PPCtfQNJewJyIuGKiGUlaLGlU0ujY2FhHQdSteaLflfHS8EHYZk7uZi1I2gY4BzipXVm/H9j6jZO7DbN27wfeHngDcLWkO4B9gRVVX1Q1K8LJ3YbZhO8HjohHImJGRMyNiLnAKmBhRIxWE+7wGPY3TJXByd2GVkRsBsbfD7wWuHT8/cCSFlYbndnUFHqHqlldtXs/cMP4/XsRk1kZXHM36yNuhrCyOLn3kLsimlmvOLmbmdWQk7uZWQ05uZuZ1ZCTu5lZDTm5d8n4xVNfRDWr3jD2QnJyNzOrISd3M7MacnI3G0LD2EwxbJzczcxqyMm9R3xh1cx6ycndzKyGnNyHmM8mzOrLyd3MrIac3IeMa+tmw8HJ3cyshpzczcxqqFByl7RA0i2S1kk6ucn0EyXdLOlGST+QtGv5oZqZWVFtk7ukacAy4CBgPnCEpPkNxX4GjETEm4DLgbPKDtTMzIorUnPfG1gXEesj4hlgOXBYvkBEXBURT2aDq4DZ5YZpZfJFVbP6K5LcZwF354Y3ZONaOQ74TrMJkhZLGpU0OjY2VjzKAeUk2v8KNDkeL+kmSddLuqbJWatZXyr1gqqkI4ER4Oxm0yPivIgYiYiRmTNnlrlos44VbHK8KCLeGBF7kpobz+lxmGaTUiS5bwTm5IZnZ+O2IOlA4FRgYUQ8XU54Zl1VpMnx0dzgy4DoYXxmk7ZtgTKrgXmSdiMl9UXA+/IFJL0Z+CKwICLuLz1Ks+5o1uS4T2MhSR8GTgSmAwc0m5GkxcBigF122aX0QM061bbmHhGbgSXAlcBa4NKIWCNpqaSFWbGzge2Ay7K2yRVdi9isxyJiWUS8Dvg4cFqLMm5ytL5SpOZORKwEVjaMOz33+cCS47Iu88VeoGCTY85y4B+7GpFZSXyHqg2zF5ocJU0nNTlucdYpaV5u8BDgth7GNzT8ZqjyFaq5m9VRRGyWNN7kOA24YLzJERiNiBXAkqyzwLPAQ8BR1UVsVpyTuw21Ak2OJ/Q8KLMSuFmmC9yebWZVc3I3M6shJ3czsxpycjczqyEndzOzGnJy74FBucA6KHGaWXtO7rYVJ3mzwefkbmZWQ07uZmY15OReEjdlmFk/8eMHSuQEb2b9wjV3M7MacnIfcj7bMKsnJ3czsxpycjfANXizunFyNzOrISd3M7MacnI3M6shJ3czsxpycjdfTDWrISd3M7MacnI3M6shJ3cbWpIWSLpF0jpJJzeZfqKkmyXdKOkHknatIk6zyXByt6EkaRqwDDgImA8cIWl+Q7GfASMR8SbgcuCs3kZpNnlO7jas9gbWRcT6iHgGWA4cli8QEVdFxJPZ4Cpgdo9jNJs0J3cbVrOAu3PDG7JxrRwHfKfVREmLJY1KGh0bGyspRLPJc3I3a0PSkcAIcHarMhFxXkSMRMTIzJkzexecWQt+WYcNq43AnNzw7GzcFiQdCJwKvCMinu5RbGZT5pq7DavVwDxJu0maDiwCVuQLSHoz8EVgYUTcX0GMZpPm5G5DKSI2A0uAK4G1wKURsUbSUkkLs2JnA9sBl0m6XtKKFrMz6ztulrGhFRErgZUN407PfT6w50GZlcQ1dzOzGnJyL4EfvGVm/cbJfYqc2M2sHzm5T8KjF/7ISd3M+lqh5F7gAUv7SfqppM2S3lN+mGZm1om2yb3gA5buAo4GLio7QDMz61yRmnuRByzdERE3As93IUargJudzAZbkeTe6QOWWvLDlfqfk7pZPfT0gqofrmRm1htFknuhByyZmVn/KJLc2z5gaVjVvQmj7t/PrM7aJvciD1iS9PuSNgDvBb4oaU03gzYzs4kVenBYgQcsrcavIDMz6xu+Q9XacvOM2eBxcrcJObGbDSYndzOzGnJyNzOrISd3M7MacnI3M6shJ3czsxpycrdC3GvGbLA4uXfACa5e/BIaqzMn9w45wdeDX0JjdefkXoATei35JTRWa07uNqxKewkN+EU01n+c3M1K4BfRWL9xcrdh5ZfQWK05uVth49ceanINwi+hsVpzci+oJgltyuqS4P0SGqu7Qi/rMGvl0Qt/xMuP+YOqw5gUv4TG6sw1d5uyQa/Fm9WRk7uZWQ05uZuZ1ZCTewtuamjP68isfzm5T8DJy8wGlZN7E07qk+P1ZtY/nNzbcMIys0Hk5G5mVkNO7mZmNeTkbmZWQ07uOW5fL4/XpVm1nNzNzGpo6JO7a5hmVkd+KmTGSX5q6vIoYLO6GPqaOzghmVn9OLmbmdWQk7uZWQ0NXXLPN8E0a45xE42Z1cHQJXczs2EwtMndNfTe8bo2672hSe7tmmPMzOpkaJK79V7jD+pEP6r+wTUrV6HkLmmBpFskrZN0cpPpL5Z0STb9Wklzyw60U/mbanyDTf/pl20xiPu2WRFtk7ukacAy4CBgPnCEpPkNxY4DHoqIPYBzgTPLDtSsbN63rc6K1Nz3BtZFxPqIeAZYDhzWUOYw4CvZ58uBP5KkyQZVpFY3XiOf6HS/X2qHw6xxGxTtftqjbdfzfdusV4o8W2YWcHdueAOwT6syEbFZ0iPAq4AH8oUkLQYWZ4OPS7ql5VKPBWBG4zzalO+W4nF0V7/EAVOJ5diG/xvHtxruLI5dC/xtFft2v2zDfokD+ieWfokD9ImJYimyb/f2wWERcR5wXtHykkYjYqSLITmOSeqXWPoljqL7dr/E2y9xQP/E0i9xQDmxFGmW2QjMyQ3PzsY1LSNpW+AVwKapBGbWA963rbaKJPfVwDxJu0maDiwCVjSUWQEclX1+D/DDiIjywjTrCu/bVlttm2WydsYlwJXANOCCiFgjaSkwGhErgC8BX5W0DniQdJCUoXATTpc5jq31SyyTjqOifXvg11sX9Ess/RIHlBCLXAkxM6sf36FqZlZDTu5mZjU0MMld0kmSQtKMipZ/tqRfSLpR0r9K2qHHy5/wNvkexTBH0lWSbpa0RtIJVcSRi2eapJ9J+naVcUxE0nuzdfW8pJZd27q9fSW9UtL3JN2W/b9ji3LPSbo++9d4cXmqMfTFox4KxHG0pLHcevhAl+K4QNL9kn7eYrokfS6L80ZJe3W0gIjo+3+krmhXAncCMyqK4Y+BbbPPZwJn9nDZ04Dbgd2B6cANwPwK1sHOwF7Z5+2BW6uIIxfPicBFwLeriqFAjL8D/BZwNTBS1fYFzgJOzj6f3Gr/BR7v0npo+x2BDwFfyD4vAi6pKI6jgc/3YN/YD9gL+HmL6QcD3wEE7Atc28n8B6Xmfi7wMaCyq78R8d2I2JwNriL1ie6VIrfJd11E3BsRP80+PwasJd3B2XOSZgOHAOdXsfyiImJtRLS+EzvpxfbNP0bhK8C7Sp5/O/3yqIe+OJYAIuLfST2wWjkM+OdIVgE7SNq56Pz7PrlLOgzYGBE3VB1LzrGkX9ReaXabfCVJdVx2yvxm4NqKQvg70g/+8xUtv0y92L47RcS92edfATu1KPcSSaOSVkkq8wegyHfc4lEPwPijHspUdF0fnjWFXC5pTpPpvTCl/aKnjx9oRdL3gdc0mXQq8DekJpFK44iIb2ZlTgU2A1/vRUz9SNJ2wDeA/x4Rj1aw/EOB+yPiOkn793r5TeJpu99UHUd+ICJCUquz4F0jYqOk3YEfSropIm4vO9Y+9y3g4oh4WtJfkM4mDqg4po71RXKPiAObjZf0RmA34Ibs7Gw28FNJe0fEr3oVRy6eo4FDgT+KrFGsR4rcJt8Tkl5ESuxfj4h/qSIG4G3AQkkHAy8BXi7paxFxZBXBtNtvCihl+04Uh6T7JO0cEfdmp/b3t5jHxuz/9ZKuJp2dlZHcO3nUw4YuPuqhbRwRkV/m+aTrFVWY0n7R180yEXFTRLw6IuZGxFzSacle3Ujs7UhaQGoGWBgRT/Z48UVuk++6rP3zS8DaiDin18sfFxGnRMTsbJ9YRHokQCWJvSS92L75xygcBWx1RiFpR0kvzj7PIP2I3lzS8vvlUQ9t42ho115IurZUhRXAn2e9ZvYFHsk1rbXX7SvCJV9dvoPqesusI7V/XZ/9+0KPl38wqXfK7aTT/SrWwdtJF7VvzK2HgyveJ/anv3vLvJtUKXkauA+4Mhv/WmBlr7Yvqe36B8BtwPeBV2bjR4Dzs89vBW4i9SC5CTiu5Bi2+o7AUlKFCdJZ2GXZsfafwO5d2ibt4vgMsCZbD1cBv92lOC4G7gWezfaR44DjgeOz6SK9TOb2bHs07W3V6p8fP2BmVkN93SxjZmaT4+RuZlZDTu5mZjXk5G5mVkNO7mZmNeTkbmZWQ07uZmY19P8BqZjW5Re/mAAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 使用'paddle.normal'实现高斯分布采样，其中'mean'为高斯分布的均值，'std'为高斯分布的标准差，'shape'为输出形状\r\n",
    "gausian_weights = paddle.normal(mean=0.0, std=1.0, shape=[10000])\r\n",
    "# 使用'paddle.uniform'实现在[min,max)范围内的均匀分布采样，其中'shape'为输出形状\r\n",
    "uniform_weights = paddle.uniform(shape=[10000], min=- 1.0, max=1.0)\r\n",
    "\r\n",
    "# 绘制两种参数分布\r\n",
    "plt.figure()\r\n",
    "plt.subplot(1,2,1)\r\n",
    "plt.title('Gausian Distribution')\r\n",
    "plt.hist(gausian_weights, bins=200, density=True, color='#f19ec2')\r\n",
    "plt.subplot(1,2,2)\r\n",
    "plt.title('Uniform Distribution')\r\n",
    "plt.hist(uniform_weights, bins=200, density=True, color='#e4007f')\r\n",
    "plt.savefig('fw-gausian-uniform.pdf')\r\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.4.2 梯度消失问题\n",
    "\n",
    "在神经网络的构建过程中，随着网络层数的增加，理论上网络的拟合能力也应该是越来越好的。但是随着网络变深，参数学习更加困难，容易出现梯度消失问题。\n",
    "\n",
    "由于Sigmoid型函数的饱和性，饱和区的导数更接近于0，误差经过每一层传递都会不断衰减。当网络层数很深时，梯度就会不停衰减，甚至消失，使得整个网络很难训练，这就是所谓的梯度消失问题。\n",
    "在深度神经网络中，减轻梯度消失问题的方法有很多种，一种简单有效的方式就是使用导数比较大的激活函数，如：ReLU。\n",
    "\n",
    "下面通过一个简单的实验观察前馈神经网络的梯度消失现象和改进方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.4.2.1 模型构建\n",
    "\n",
    "定义一个前馈神经网络，包含4个隐藏层和1个输出层，通过传入的参数指定激活函数。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 定义多层前馈神经网络\n",
    "class Model_MLP_L5(paddle.nn.Layer):\n",
    "    def __init__(self, input_size, output_size, act='sigmoid', w_init=Normal(mean=0.0, std=0.01), b_init=Constant(value=1.0)):\n",
    "        super(Model_MLP_L5, self).__init__()\n",
    "        self.fc1 = paddle.nn.Linear(input_size, 3)\n",
    "        self.fc2 = paddle.nn.Linear(3, 3)\n",
    "        self.fc3 = paddle.nn.Linear(3, 3)\n",
    "        self.fc4 = paddle.nn.Linear(3, 3)\n",
    "        self.fc5 = paddle.nn.Linear(3, output_size)\n",
    "        # 定义网络使用的激活函数\n",
    "        if act == 'sigmoid':\n",
    "            self.act = F.sigmoid\n",
    "        elif act == 'relu':\n",
    "            self.act = F.relu\n",
    "        elif act == 'lrelu':\n",
    "            self.act = F.leaky_relu\n",
    "        else:\n",
    "            raise ValueError(\"Please enter sigmoid relu or lrelu!\")\n",
    "        # 初始化线性层权重和偏置参数\n",
    "        self.init_weights(w_init, b_init)\n",
    "\n",
    "    # 初始化线性层权重和偏置参数\n",
    "    def init_weights(self, w_init, b_init):\n",
    "        # 使用'named_sublayers'遍历所有网络层\n",
    "        for n, m in self.named_sublayers():\n",
    "            # 如果是线性层，则使用指定方式进行参数初始化\n",
    "            if isinstance(m, nn.Linear):\n",
    "                w_init(m.weight)\n",
    "                b_init(m.bias)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        outputs = self.fc1(inputs)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc2(outputs)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc3(outputs)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc4(outputs)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc5(outputs)\n",
    "        outputs = F.sigmoid(outputs)\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.4.2.2 使用Sigmoid型函数进行训练\n",
    "\n",
    "使用Sigmoid型函数作为激活函数，为了便于观察梯度消失现象，只进行一轮网络优化。代码实现如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "定义梯度打印函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def print_grads(runner):\n",
    "    # 打印每一层的权重的模\n",
    "    print('The gradient of the Layers：')\n",
    "    for item in runner.model.sublayers():\n",
    "        if len(item.parameters())==2:\n",
    "            print(item.full_name(), paddle.norm(item.parameters()[0].grad, p=2.).numpy()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "paddle.seed(102)\n",
    "# 学习率大小\n",
    "lr = 0.01\n",
    "\n",
    "# 定义网络，激活函数使用sigmoid\n",
    "model =  Model_MLP_L5(input_size=2, output_size=1, act='sigmoid')\n",
    "\n",
    "# 定义优化器\n",
    "optimizer = paddle.optimizer.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "# 定义损失函数，使用交叉熵损失函数\n",
    "loss_fn = F.binary_cross_entropy\n",
    "\n",
    "# 定义评价指标\n",
    "metric = accuracy\n",
    "\n",
    "# 指定梯度打印函数\n",
    "custom_print_log=print_grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "实例化RunnerV2_2类，并传入训练配置。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 实例化Runner类\n",
    "runner = RunnerV2_2(model, optimizer, metric, loss_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "模型训练，打印网络每层梯度值的$\\ell_2$范数。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The gradient of the Layers：\n",
      "linear_4 6.9098754e-11\n",
      "linear_5 1.648972e-08\n",
      "linear_6 4.8029387e-06\n",
      "linear_7 0.0015030989\n",
      "linear_8 0.29127562\n",
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.46250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/varbase_patch_methods.py:392: UserWarning: \u001b[93m\n",
      "Warning:\n",
      "tensor.grad will return the tensor value of the gradient. This is an incompatible upgrade for tensor.grad API.  It's return type changes from numpy.ndarray in version 2.0 to paddle.Tensor in version 2.1.0.  If you want to get the numpy value of the gradient, you can use :code:`x.grad.numpy()` \u001b[0m\n",
      "  warnings.warn(warning_msg)\n"
     ]
    }
   ],
   "source": [
    "# 启动训练\n",
    "runner.train([X_train, y_train], [X_dev, y_dev], \n",
    "            num_epochs=1, log_epochs=None, \n",
    "            save_path=\"best_model.pdparams\", \n",
    "            custom_print_log=custom_print_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "观察实验结果可以发现，梯度经过每一个神经层的传递都会不断衰减，最终传递到第一个神经层时，梯度几乎完全消失。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.4.2.3 使用ReLU函数进行模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The gradient of the Layers：\n",
      "linear_9 4.6598142e-08\n",
      "linear_10 3.0365838e-06\n",
      "linear_11 0.0001731529\n",
      "linear_12 0.010652662\n",
      "linear_13 0.40262264\n",
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.46250\n"
     ]
    }
   ],
   "source": [
    "paddle.seed(102)\n",
    "lr = 0.01  # 学习率大小\n",
    "\n",
    "# 定义网络，激活函数使用relu\n",
    "model =  Model_MLP_L5(input_size=2, output_size=1, act='relu')\n",
    "\n",
    "# 定义优化器\n",
    "optimizer = paddle.optimizer.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "# 定义损失函数\n",
    "# 定义损失函数，这里使用交叉熵损失函数\n",
    "loss_fn = F.binary_cross_entropy\n",
    "\n",
    "# 定义评估指标\n",
    "metric = accuracy\n",
    "\n",
    "# 实例化Runner\n",
    "runner = RunnerV2_2(model, optimizer, metric, loss_fn)\n",
    "\n",
    "# 启动训练\n",
    "runner.train([X_train, y_train], [X_dev, y_dev], \n",
    "            num_epochs=1, log_epochs=None, \n",
    "            save_path=\"best_model.pdparams\", \n",
    "            custom_print_log=custom_print_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**图4.4** 展示了使用不同激活函数时，网络每层梯度值的$\\ell_2$范数情况。从结果可以看到，5层的全连接前馈神经网络使用Sigmoid型函数作为激活函数时，梯度经过每一个神经层的传递都会不断衰减，最终传递到第一个神经层时，梯度几乎完全消失。改为ReLU激活函数后，梯度消失现象得到了缓解，每一层的参数都具有梯度值。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/d1fe3501bdbe47c2be309151c4aa2cf8419b561a64184e468e89c7d3de4f480d\" width=\"600\"  ></center>\n",
    "<center><br>图4.4：网络每层梯度的L2范数变化趋势</br></center>\n",
    "<br></br> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.4.3  死亡 ReLU 问题\n",
    "\n",
    "ReLU激活函数可以一定程度上改善梯度消失问题，但是ReLU函数在某些情况下容易出现死亡 ReLU问题，使得网络难以训练。这是由于当$x<0$时，ReLU函数的输出恒为0。在训练过程中，如果参数在一次不恰当的更新后，某个ReLU神经元在所有训练数据上都不能被激活（即输出为0），那么这个神经元自身参数的梯度永远都会是0，在以后的训练过程中永远都不能被激活。而一种简单有效的优化方式就是将激活函数更换为Leaky ReLU、ELU等ReLU的变种。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.4.3.1 使用ReLU进行模型训练\n",
    "\n",
    "使用第4.4.2节中定义的多层全连接前馈网络进行实验，使用ReLU作为激活函数，观察死亡ReLU现象和优化方法。当神经层的偏置被初始化为一个相对于权重较大的负值时，可以想像，输入经过神经层的处理，最终的输出会为负值，从而导致死亡ReLU现象。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 定义网络，并使用较大的负值来初始化偏置\n",
    "model =  Model_MLP_L5(input_size=2, output_size=1, act='relu', b_init=Constant(value=-8.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "实例化RunnerV2类，启动模型训练，打印网络每层梯度值的$\\ell_2$范数。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The gradient of the Layers：\n",
      "linear_14 0.0\n",
      "linear_15 0.0\n",
      "linear_16 0.0\n",
      "linear_17 0.0\n",
      "linear_18 0.0\n",
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.53750\n"
     ]
    }
   ],
   "source": [
    "# 实例化Runner类\n",
    "runner = RunnerV2_2(model, optimizer, metric, loss_fn)\n",
    "\n",
    "# 启动训练\n",
    "runner.train([X_train, y_train], [X_dev, y_dev], \n",
    "            num_epochs=1, log_epochs=0, \n",
    "            save_path=\"best_model.pdparams\", \n",
    "            custom_print_log=custom_print_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从输出结果可以发现，使用 ReLU 作为激活函数，当满足条件时，会发生死亡ReLU问题，网络训练过程中 ReLU 神经元的梯度始终为0，参数无法更新。\n",
    "\n",
    "针对死亡ReLU问题，一种简单有效的优化方式就是将激活函数更换为Leaky ReLU、ELU等ReLU 的变种。接下来，观察将激活函数更换为 Leaky ReLU时的梯度情况。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.4.3.2 使用Leaky ReLU进行模型训练\n",
    "\n",
    "将激活函数更换为Leaky ReLU进行模型训练，观察梯度情况。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The gradient of the Layers：\n",
      "linear_19 2.311491e-16\n",
      "linear_20 1.8818199e-13\n",
      "linear_21 1.6921159e-09\n",
      "linear_22 1.1889538e-05\n",
      "linear_23 0.07010835\n",
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.53750\n",
      "[Train] epoch: 0/1, loss: 4.050660133361816\n"
     ]
    }
   ],
   "source": [
    "# 重新定义网络，使用Leaky ReLU激活函数\n",
    "model =  Model_MLP_L5(input_size=2, output_size=1, act='lrelu', b_init=Constant(value=-8.0))\n",
    "\n",
    "# 实例化Runner类\n",
    "runner = RunnerV2_2(model, optimizer, metric, loss_fn)\n",
    "\n",
    "# 启动训练\n",
    "runner.train([X_train, y_train], [X_dev, y_dev], \n",
    "            num_epochs=1, log_epochps=None, \n",
    "            save_path=\"best_model.pdparams\", \n",
    "            custom_print_log=custom_print_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从输出结果可以看到，将激活函数更换为Leaky ReLU后，死亡ReLU问题得到了改善，梯度恢复正常，参数也可以正常更新。但是由于 Leaky ReLU 中，$\\mathcal{x<0}$ 时的斜率默认只有0.01，所以反向传播时，随着网络层数的加深，梯度值越来越小。如果想要改善这一现象，将 Leaky ReLU 中，$\\mathcal{x<0}$ 时的斜率调大即可。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
